Dimensionality-Reduction-PCA-and-Convolutional Neural Networks on Mnist Dataset-Kaggle Competition Notebook
By: Rohan Paul
Tags: Statistics, Python, Machine Learning, Neural Networks, Data Science

Don’t confuse PCA with linear regression. PCA is to minimize the distance of the orthogonal projection, while linear regression is to minimize the distance on the y-axis.

Source of above image

Why is minimizing squared residuals equivalent to maximizing variance?

To ans this question I could not find a better explanations than has been described here in this blog

Where w is the projection of each datapoint onto the top principal component and the vector c is the top principal component

Important Note — PCA assumes that the dataset is centered around the origin.

As we will see, Scikit-Learn’s PCA classes take care of centering the data for you. If you implement PCA yourself, or if you use other libraries, don’t forget to center the data first

Now I will Calculate the PCA for MNIST data quite manually using the steps described earlier, which are

Step-A => Standardize the Columns

Step-B => Build the Covariance-Matrix S which is $(X^T * X)$

Step-C => Get the EigneVectors of this Covariance-Matrix S. These will be sorted from largest λ and downwards. And my u1 is the Eigenvector v1 which is the largest Eigenvector corresponding to the largest Eigenvalue

is the Eigenvector which is the largest Eigenvector corresponding to the largest Eigenvalue Step-D => Now I need to project the original data sample on the plane formed by two principle eigenvectors by vector-vector multiplication.

The training dataset is 42,000 small square 28×28 pixel grayscale images of handwritten single digits between 0 and 9. The task is to classify a given image of a handwritten digit into one of 10 classes representing integer values from 0 to 9, inclusively.

import numpy as np

import pandas as pd

import seaborn as sns

sns.set_style("dark")

import matplotlib.pyplot as plt

from sklearn.ensemble import RandomForestClassifier

from sklearn.model_selection import train_test_split

from sklearn.neighbors import KNeighborsClassifier

from sklearn import decomposition

from sklearn.preprocessing import StandardScaler # In my local Machine, I ah only taking fewer rows for faster local training and development

# In Kaggle comment-out below line

train_df_org = pd.read_csv('kaggle/input/digit-recognizer/train.csv', nrows=1000)

test_df_org = pd.read_csv('kaggle/input/digit-recognizer/test.csv', nrows=1000)



# Instead, uncomment the below one, where the whole dataset is taken for training

# train_df_org = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')

# test_df_org = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')



print("train_df_org shape ", train_df_org.shape) # (1000, 785)

print("test_df_org shape ", test_df_org.shape)



train_df_org.head() train_df_org shape (1000, 785)

test_df_org shape (1000, 784)

# In test df there should not be any label column.

# lets see it.

test_df_org.head()

In below the labels variable contains the labels and then from the ‘train_df_org’ (which contains all the actual image pixel) I will drop ‘label’ column

labels = train_df_org['label']

labels.shape (1000,) train_df_for_pca = train_df_org.drop(['label'], axis=1)

train_df_for_pca.shape

# In pandas, drop( ) function is used to remove column(s).axis=1 tells Python that you want to apply function on columns instead of rows. (1000, 784)

Step-A: Data-preprocessing: Standardizing the train dataframe

(x - mean) / s.d

standardized_data = StandardScaler().fit_transform(train_df_for_pca)

print(standardized_data.shape) (1000, 784)

Step-B: Now build the c-variance matrix which is :

$A^T * A$

sample_data = standardized_data



# Matrix multiplication with numpy

covariance_matrix = np.matmul(sample_data.T, sample_data)



# As the sample_data has 784 columns, so the co-variance matrix shape

# should be 784 * 784

print('Shape of Co-variance matrix = ', covariance_matrix.shape) Shape of Co-variance matrix = (784, 784)

Step-C => Get the EigneVectors of this Covariance-Matrix S for projecting onto a 2-D space. These Eigenvectors will be sorted by the value of λ.

from scipy.linalg import eigh



# eigh function will give me the eigenvalues in ascending order with zero-index

# subset_by_index is an optional and iterable parameter

# If provided, this two-element iterable defines the start and the end indices

# of the desired eigenvalues (ascending order and 0-indexed).

# To return only the second smallest to fifth smallest eigenvalues, [1, 4] is used.

# [n-3, n-1] returns the largest three.

# Only available with “evr”, “evx”, and “gvx” drivers. The entries are directly converted to integers via int().



# Needed to comment out below line containing 'subset_by_index' parameter, as it would fail

# in Kaggle, probably because of the driver requirement.

# eigenvalues, eigenvectors = eigh(covariance_matrix, subset_by_index=(782, 783) )



# instead using 'eigvals' which is deprecated per doc

eigenvalues, eigenvectors = eigh(covariance_matrix, eigvals=(782, 783) )

print('Shape of Eigenvectors ', eigenvectors.shape )



# In above I am only passing 782 and 783 as eigvalues becuase

# Our target is to reduce the 784-dimension data to 2-dimension data and plot it.

# So we need the top 2 eigvalues. Shape of Eigenvectors (784, 2)

So you have the principal components. They are eigenvectors of the covariance matrix $X^T X$.

A way to retrieve the eigenvalues from there is to apply this matrix to each principal components and project the results onto the component. Let v_1 be the first principal component and lambda_1 the associated eigenvalue. We have:

and thus:

since

. (x, y) the scalar product of vectors x and y.

# now converting the eigenvectors into (2, d) shape for ease of computation further ahead

eigenvectors = eigenvectors.T

eigenvectors[0] represents the eigenvector corresponding to 1st Principal Component

eigenvectors1 represents the eigenvector corresponding to 2nd Principal Component

Now finally our target which was to reduce the 784-dimension data to 2-dimension data and plot it.

Step*D => Now I need to project the original data sample on the plane formed by two principle eigenvectors by vector-vector multiplication.

Simplest Formulate to Refer for Vector Projection

Source

projected_vec = np.matmul(eigenvectors, sample_data.T)

projected_vec.shape

# Now we will see this is a 2-dimensional data (2, 1000)

Now I will use vstack() function to Stack arrays in sequence vertically (row wise). This function makes most sense for arrays with up to 3 dimensions. For instance, for pixel-data with a height (first axis), width (second axis), and r/g/b channels (third axis).

a = np.array([1, 2, 3])

b = np.array([2, 3, 4])

np.vstack((a,b))

array([[1, 2, 3],

[2, 3, 4]]) # appending label to the 2-d projected data

projected_vec = np.vstack((projected_vec, labels)).T

projected_vec.shape (1000, 3) # The projected_vec at this point is still in array form and so we can not apply

# head() function to it directly and also for plotting purpose, I need to make a dataframe from it.

# creating a new dataframe from plotting the labeled points

pca_dataframe = pd.DataFrame(data=projected_vec, columns=('1st_principal_comp', "2nd_principal_comp", 'labels') )

pca_dataframe.head()

So in above each row is x_i which was originally 784 dimensions and now they are in 2-dimensions by applying PCA.

sns.FacetGrid(pca_dataframe, hue='labels', size=6).map(plt.scatter, '1st_principal_comp', "2nd_principal_comp").add_legend()

plt.show() /home/paul/anaconda3/lib/python3.7/site-packages/seaborn/axisgrid.py:316: UserWarning: The `size` parameter has been renamed to `height`; please update your code.

warnings.warn(msg, UserWarning)

And we can see some degree of classifications among the digits. For example, all zeros are on the top left corner. All the sixes (oranges) are towards the bottom between 0 and 4 on the x-axis.

Now deriving PCA with scikit-learn’s PCA function

pca = decomposition.PCA()

pca.n_components = 2

pca_data_with_scikit = pca.fit_transform(standardized_data)

pca_data_with_scikit.shape (1000, 2) pca_data_with_scikit = np.vstack((pca_data_with_scikit.T, labels)).T

pca_data_with_scikit.shape (1000, 3) df_PCA_scikit = pd.DataFrame(data=pca_data_with_scikit, columns=('f1_PC', 'f2_PC', 'labels'))

df_PCA_scikit.head()

sns.FacetGrid(df_PCA_scikit, hue='labels', size=6).map(plt.scatter, 'f1_PC', 'f2_PC').add_legend()

plt.show() /home/paul/anaconda3/lib/python3.7/site-packages/seaborn/axisgrid.py:316: UserWarning: The `size` parameter has been renamed to `height`; please update your code.

warnings.warn(msg, UserWarning)

And I am getting almost similar plotting that we got with the manual steps to calculate Principle Components. Main difference with sickit-learn the axes are reversed.

Plot to show how Variance Retained increases as we increase the Number of Principle components

print('Shape of standardized_data ', standardized_data.shape)



pca.n_components = 784

pca_data_scikit_2 = pca.fit_transform(standardized_data)

percent_variance_retained = pca.explained_variance_ / np.sum(pca.explained_variance_)

cumulative_variance_retained = np.cumsum(percent_variance_retained) Shape of standardized_data (1000, 784) plt.figure(1, figsize=(10, 6))

plt.clf()

plt.plot(cumulative_variance_retained, linewidth=2)

plt.axis('tight')

plt.grid()

plt.xlabel('Number of Components')

plt.ylabel('Cumulative variance Retained')

plt.show()

So above was all about Dimensionality Reduction and PCA

Now for the Competition I will run a CNN Model on the MNIST dataset

import tensorflow as tf



from tensorflow.keras import layers

from tensorflow.keras.models import Model

from tensorflow.keras import metrics

from tensorflow.keras import backend as K

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Dense, Dropout, Lambda, Flatten, BatchNormalization

from tensorflow.keras.layers import Conv2D, MaxPool2D, AvgPool2D

from tensorflow.keras.optimizers import Adadelta

from keras.utils.np_utils import to_categorical

from tensorflow.keras.preprocessing.image import ImageDataGenerator

from tensorflow.keras.callbacks import ReduceLROnPlateau

from tensorflow.keras.callbacks import LearningRateScheduler # train_df_org = pd.read_csv('kaggle/input/digit-recognizer/train.csv', nrows=1000)



# train_df_org = pd.read_csv('/kaggle/input/digit-recognizer/train.csv', nrows=1000)



# Now dataframe with only 'label' column

mnist_train_label = train_df_org.loc[:, "label"]

# print(mnist_train_label.shape) # (48000,)



mnist_train_df = train_df_org.loc[:, "pixel0":]

# print(mnist_train_df.shape) # (48000, 784) # Now convert the dataframes into numpy arrays

mnist_train_array = np.array(mnist_train_df)

mnist_train_label_array = np.array(mnist_train_label)



# Now need to do a little reshaping to get the structure which the model will expect to receive going forward

mnist_train_array = mnist_train_array.reshape(mnist_train_array.shape[0], 28, 28, 1)

print(mnist_train_array.shape, mnist_train_label_array.shape)

print(mnist_train_label_array) (1000, 28, 28, 1) (1000,)

[1 0 1 4 0 0 7 3 5 3 8 9 1 3 3 1 2 0 7 5 8 6 2 0 2 3 6 9 9 7 8 9 4 9 2 1 3

1 1 4 9 1 4 4 2 6 3 7 7 4 7 5 1 9 0 2 2 3 9 1 1 1 5 0 6 3 4 8 1 0 3 9 6 2

6 4 7 1 4 1 5 4 8 9 2 9 9 8 9 6 3 6 4 6 2 9 1 2 0 5 9 2 7 7 2 8 8 5 0 6 0

0 2 9 0 4 7 7 1 5 7 9 4 6 1 5 7 6 5 0 4 8 7 6 1 8 7 3 7 3 1 0 3 4 5 4 0 5

4 0 3 5 1 0 8 3 7 0 9 6 6 9 5 4 6 9 3 5 4 2 4 8 7 7 5 8 8 8 2 6 9 3 1 0 4

1 5 9 0 6 2 1 3 0 6 0 0 8 3 2 0 0 6 0 0 4 7 2 7 1 9 9 3 9 8 4 6 6 5 3 8 1

8 7 1 3 7 6 3 6 3 6 3 2 3 2 2 7 9 2 3 2 7 5 5 8 8 2 0 1 4 0 6 3 7 1 1 1 4

7 0 2 9 2 0 5 6 0 8 9 6 2 0 0 7 2 0 4 2 0 9 1 6 9 3 0 0 2 0 6 8 4 0 7 2 1

9 5 2 4 8 5 2 9 7 9 2 9 7 4 9 3 2 7 3 6 3 6 8 8 3 7 0 9 2 7 9 0 5 4 5 8 4

3 3 1 7 8 9 7 6 2 1 7 0 5 6 5 2 9 5 4 6 2 2 2 9 0 7 7 2 2 6 3 4 2 0 5 9 6

2 1 9 0 6 0 4 8 4 3 1 5 4 2 9 5 7 3 1 5 4 5 3 7 3 8 6 2 4 6 1 1 4 0 0 5 8

6 7 4 2 8 0 2 5 4 8 3 0 6 4 8 6 4 1 8 1 5 4 9 4 3 2 0 5 0 7 9 2 9 8 9 6 5

2 4 4 6 4 8 4 1 7 5 8 9 5 9 3 2 5 8 2 2 7 2 8 4 1 9 3 6 0 2 2 9 1 2 7 2 1

3 4 9 1 8 0 2 2 3 4 1 3 7 4 1 4 1 5 9 6 9 0 5 7 6 8 2 0 7 3 5 8 2 8 2 4 8

5 8 9 7 1 2 4 5 5 1 8 1 4 4 6 5 8 9 2 3 0 5 1 4 0 5 1 2 9 2 4 1 6 8 0 4 9

0 0 5 9 2 3 5 9 4 4 3 9 2 3 5 6 5 2 7 2 4 2 4 7 2 5 3 7 6 1 0 7 5 4 5 1 6

9 7 1 6 3 3 1 2 2 0 5 0 6 8 3 6 7 7 3 8 1 7 9 3 9 2 8 3 7 4 1 2 3 6 5 0 1

8 6 9 2 1 6 0 2 8 0 8 8 9 1 2 2 1 4 8 1 4 4 5 1 8 7 7 9 7 0 6 9 4 5 6 2 5

7 4 7 2 3 0 8 4 8 0 0 9 7 7 9 8 2 1 6 5 5 1 1 9 7 7 8 6 4 7 5 3 1 6 4 5 7

4 1 8 3 5 1 7 1 1 8 6 4 3 8 3 1 2 8 9 0 9 1 2 3 3 0 3 0 2 0 3 3 8 3 5 7 0

5 9 0 5 9 1 5 1 1 2 6 5 5 4 5 1 6 0 2 2 8 0 7 1 0 8 5 6 3 2 9 4 3 6 0 3 4

1 5 9 3 0 5 0 6 2 7 6 6 6 9 6 7 8 2 0 6 0 8 9 5 3 6 7 4 3 9 7 2 0 4 7 2 2

8 2 7 0 4 0 5 2 8 7 7 9 1 4 0 1 1 2 3 6 2 0 6 6 1 9 4 5 2 7 7 8 9 5 8 3 8

5 6 2 0 9 7 1 8 2 6 9 8 4 9 4 1 3 8 4 0 7 7 3 7 6 6 8 8 2 7 0 4 3 7 7 0 8

4 7 4 0 6 9 8 6 0 1 6 4 5 2 7 3 6 2 2 9 2 7 4 8 7 2 9 5 3 4 8 0 4 4 6 5 6

1 2 2 8 4 5 7 8 0 6 8 9 1 7 7 2 6 3 9 9 1 0 4 2 5 4 4 9 2 6 7 2 8 3 3 2 7

0 4 7 0 7 7 8 1 7 3 7 8 0 1 0 2 9 7 6 2 2 6 9 0 6 8 8 9 6 3 5 0 2 2 5 9 6

4]

CNN Model Building

Training a neural network with one input layer, one hidden layer and one output layer for learning the digits in images.

Encoding the labels and making them as the class value and finally converting them as categorical values.

Using the method to_categorical(), a numpy array (or) a vector which has integers that represent different categories, can be converted into a numpy array (or) a matrix which has binary values and has columns equal to the number of categories in the data. For example, label ‘3’ should be converted to a vector [0, 0, 0, 1, 0, 0, 0, 0, 0, 0] for model building. This function returns a matrix of binary values (either ‘1’ or ‘0’). It has number of rows equal to the length of the input vector and number of columns equal to the number of classes.

So the whole mnist_train_label would look like below

[[0. 1. 0. ... 0. 0. 0.]

[1. 0. 0. ... 0. 0. 0.]

[0. 1. 0. ... 0. 0. 0.]

...

[0. 0. 0. ... 0. 0. 1.]

[0. 0. 0. ... 0. 0. 0.]

[0. 0. 0. ... 0. 0. 0.]] num_of_digit_classes = mnist_train_label_array.max() - mnist_train_label_array.min() + 1

mnist_train_label_array = to_categorical(mnist_train_label_array, num_classes=num_of_digit_classes)

print('Shape of ytrain after encoding and converting to categorical values ', mnist_train_label_array.shape)

print(mnist_train_label_array) Shape of ytrain after encoding and converting to categorical values (1000, 10)

[[0. 1. 0. ... 0. 0. 0.]

[1. 0. 0. ... 0. 0. 0.]

[0. 1. 0. ... 0. 0. 0.]

...

[0. 0. 0. ... 0. 0. 1.]

[0. 0. 0. ... 0. 0. 0.]

[0. 0. 0. ... 0. 0. 0.]] def run_model(input_shape=(28, 28, 1)):

model = Sequential()

model.add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = input_shape))

model.add(BatchNormalization())

model.add(Conv2D(32, kernel_size = 3, activation='relu'))

model.add(BatchNormalization())

model.add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))

model.add(BatchNormalization())

model.add(Dropout(0.4))



model.add(Conv2D(64, kernel_size = 3, activation='relu'))

model.add(BatchNormalization())

model.add(Conv2D(64, kernel_size = 3, activation='relu'))

model.add(BatchNormalization())

model.add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))

model.add(BatchNormalization())

model.add(Dropout(0.4))



model.add(Conv2D(128, kernel_size = 4, activation='relu'))

model.add(BatchNormalization())

model.add(Flatten())

model.add(Dropout(0.4))

model.add(Dense(10, activation='softmax'))

return model



def run_model_compilation(model, optimizer='adam', loss='categorical_crossentropy'):

model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])



def run_model_training(model, train, test, epochs, split):

history=model.fit(train, test, shuffle=True, epochs=epochs, validation_split=split )

return history

Neural Network Model’s parameters explained

Sequential() -

A sequential model, which is a linear stack of layers as the name suggests, allows you to create models layer-by-layer in a step-by-step fashion. Keras Sequential API is by far the easiest way to get up and running with Keras, but it is s also the most limited — you cannot create models that:

Share layers Have branches (at least not easily) Have multiple inputs Have multiple outputs

The model needs to know what input shape it should expect. For this reason, the first layer in a sequential model (and only the first, because following layers can do automatic shape inference) needs to receive information about its input shape.

As illustrated in the example above, this is done by passing an input_shape argument to the first layer. This is a list of integers or NULL entries, where NULL indicates that any positive integer may be expected. In input_shape, the batch dimension is not included.

If you ever need to specify a fixed batch size for your inputs (this is useful for stateful recurrent networks), you can pass a batch_size argument to a layer. If you pass both batch_size=32 and input_shape=c(6, 8) to a layer, it will then expect every batch of inputs to have the batch shape (32, 6, 8).

The activation parameter

The activation parameter to the Conv2D class is simply a convenience parameter, allowing you to supply a string specifying the name of the activation function you want to apply after performing the convolution.

ReLU (Rectified Linear Unit) Activation Function

The ReLU is the most used activation function in the world right now.Since, it is used in almost all the convolutional neural networks or deep learning.

The ReLU function is f(x)=max(0,x).

As you can see, the ReLU is half rectified (from bottom). f(z) is zero when z is less than zero and f(z) is equal to z when z is above or equal to zero. Range: [ 0 to infinity) The function and its derivative both are monotonic. But the issue is that all the negative values become zero immediately which decreases the ability of the model to fit or train from the data properly. That means any negative input given to the ReLU activation function turns the value into zero immediately in the graph, which in turns affects the resulting graph by not mapping the negative values appropriately.

One way ReLUs improve neural networks is by speeding up training. The gradient computation is very simple (either 0 or 1 depending on the sign of x). Also, the computational step of a ReLU is easy: any negative elements are set to 0.0 — no exponentials, no multiplication or division operations.

Gradients of logistic and hyperbolic tangent networks are smaller than the positive portion of the ReLU. This means that the positive portion is updated more rapidly as training progresses. However, this comes at a cost. The 0 gradient on the left-hand side is has its own problem, called “dead neurons,” in which a gradient update sets the incoming values to a ReLU such that the output is always zero; modified ReLU units such as ELU (or Leaky ReLU, or PReLU, etc.) can ameliorate this.

BatchNormalization in Keras -

The layer will transform inputs so that they are standardized, meaning that they will have a mean of zero and a standard deviation of one.

During training, the layer will keep track of statistics for each input variable and use them to standardize the data.

Further, the standardized output can be scaled using the learned parameters of Beta and Gamma that define the new mean and standard deviation for the output of the transform. The layer can be configured to control whether these additional parameters will be used or not via the “center” and “scale” attributes respectively. By default, they are enabled.

The statistics used to perform the standardization, e.g. the mean and standard deviation of each variable, are updated for each mini batch and a running average is maintained.

A “momentum” argument allows you to control how much of the statistics from the previous mini batch to include when the update is calculated. By default, this is kept high with a value of 0.99. This can be set to 0.0 to only use statistics from the current mini-batch, as described in the original paper.

Keras Conv2D is a 2D Convolution Layer

This layer creates a convolution kernel that is wind with layers input which helps produce a tensor of outputs.

Kernel: In image processing kernel is a convolution matrix or masks which can be used for blurring, sharpening, embossing, edge detection, and more by doing a convolution between a kernel and an image.

Filters

The first required Conv2D parameter is the number of filters that the convolutional layer will learn.

The Keras Conv2D parameter, filters determines the number of kernels to convolve with the input volume. Each of these operations produces a 2D activation map.

Layers early in the network architecture (i.e., closer to the actual input image) learn fewer convolutional filters while layers deeper in the network (i.e., closer to the output predictions) will learn more filters.

Conv2D layers in between will learn more filters than the early Conv2D layers but fewer filters than the layers closer to the output. Let’s go ahead and take a look at an example:

So filter is a mandatory Conv2D parameter. It is an integer value and also determines the number of output filters in the convolution.

model.add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = input_shape))

In above line we are learning a total of 32 filters. As far as choosing the appropriate value for no. of filters, it is always recommended to use powers of 2 as the values.

Read more on Keras Conv2D here — Some excerpts from this

What is a 2D Convolution Layer, the Convolution Kernel and its Role in CNN Image Classification

Briefly, some background. A convolution layer “scans” A source image with a filter of, for example, 5×5 pixels, to extract features which may be important for classification. This filter is also called the convolution kernel. The kernel also contains weights, which are tuned in the training of the model to achieve the most accurate predictions.

In a 5×5 kernel, for each 5×5 pixel region, the model computes the dot products between the image pixel values and the weights defined in the filter.

A 2D convolution layer means that the input of the convolution operation is three-dimensional, for example, a color image which has a value for each pixel across three layers: red, blue and green. However, it is called a “2D convolution” because the movement of the filter across the image happens in two dimensions. The filter is run across the image three times, once for each of the three layers.

After the convolution ends, the features are downsampled, and then the same convolutional structure repeats again. At first, the convolution identifies features in the original image (for example in a cat, the body, legs, tail, head), then it identifies sub-features within smaller parts of the image (for example, within the head, the ears, whiskers, eyes). Eventually, this process is meant to identify the essential features that can help classify the image.

The kernel_size.

The kernel_size must be an odd integer.

Typical values for kernel_size include: (1, 1) , (3, 3) , (5, 5) , (7, 7) . It’s rare to see kernel sizes larger than 7×7.

So, when do you use each?

If your input images are greater than 128×128 you may choose to use a kernel size > 3 to help (1) learn larger spatial filters and (2) to help reduce volume size.

strides

The strides parameter is a 2-tuple of integers, specifying the “step” of the convolution along the x and y axis of the input volume.

The strides value defaults to (1, 1) , implying that:

A given convolutional filter is applied to the current location of the input volume The filter takes a 1-pixel step to the right and again the filter is applied to the input volume This process is performed until we reach the far-right border of the volume in which we move our filter one pixel down and then start again from the far left Typically you’ll leave the strides parameter with the default (1, 1) value; however, you may occasionally increase it to (2, 2) to help reduce the size of the output volume (since the step size of the filter is larger).

Typically you’ll see strides of 2×2 as a replacement to max pooling

padding

The padding parameter to the Keras Conv2D class can take on one of two values: valid or same .

With the valid parameter the input volume is not zero-padded and the spatial dimensions are allowed to reduce via the natural application of convolution.

If you instead want to preserve the spatial dimensions of the volume such that the output volume size matches the input volume size, then you would want to supply a value of same for the padding :

Dropout

From Tensorflow Docs

The parameter rate - Float between 0 and 1. Fraction of the input units to drop.

The Dropout layer randomly sets input units to 0 with a frequency of rate at each step during training time, which helps prevent overfitting. Inputs not set to 0 are scaled up by 1/(1 — rate) such that the sum over all inputs is unchanged.

Note that the Dropout layer only applies when training is set to True such that no values are dropped during inference. When using model.fit, training will be appropriately set to True automatically, and in other contexts, you can set the kwarg explicitly to True when calling the layer.

The following taken from this very informative blog

Neural networks have hidden layers in between their input and output layers, these hidden layers have neurons embedded within them, and it’s the weights within the neurons along with the interconnection between neurons is what enables the neural network system to learn. The general idea in Neural networks is that the more neurons and layers within a neural network architecture, the greater the representational power it has. This increase in representational power means that the neural network can fit more complex functions and generalize well to training data. Simply kept, there are more configurations for the interconnections between the neurons within the neural network layers.

The disadvantage of utilizing deeper neural networks is that they are highly prone to overfitting. Overfitting is a common problem that is defined as the inability for a trained machine learning model to generalized well to unseen data, but the same model performs well on the data it was trained on. The primary purpose of dropout is to minimize the effect of overfitting within a trained network. Dropout technique works by randomly reducing the number of interconnecting neurons within a neural network. At every training step, each neuron has a chance of being left out, or rather, dropped out of the collated contribution from connected neurons.

This technique minimizes overfitting because each neuron becomes independently sufficient, in the sense that the neurons within the layers learn weight values that are not based on the cooperation of its neighbouring neurons. Supposedly you trained 7,000 different neural network architecture, to select the best one you simply take the average of all 7,000 trained neural network.

Well, the dropout technique actually mimics this scenario. If the probability of a neuron getting dropped out in a training step is set to 0.5; we are actually training a variety of different network at each training step as it’s highly impossible that the same neurons are excluded at any two training steps. Therefore a neural network that has been trained utilizing the dropout technique is an average of all the different neurons connection combinations that have occurred at each training step.

Training and evaluating the model

After the model architecture is defined and compiled, the model needs to be trained with training data to be able to recognize the handwritten digits. Here, one epoch means one forward and one backward pass of all the training samples. Batch size implies number of training samples in one forward/backward pass.

# Now actually train the model implementing all the functions above

cnn_model = run_model((28,28, 1))

run_model_compilation(cnn_model, 'adam', 'categorical_crossentropy')



# Taining the model specifying epocs

model_history = run_model_training(cnn_model, mnist_train_array, mnist_train_label_array, 75, 0.2) Epoch 1/2

2/25 [=>............................] - ETA: 0s - loss: 3.3336 - accuracy: 0.1719WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0098s vs `on_train_batch_end` time: 0.0746s). Check your callbacks.

25/25 [==============================] - 3s 110ms/step - loss: 2.3461 - accuracy: 0.3237 - val_loss: 1.9389 - val_accuracy: 0.3650

Epoch 2/2

25/25 [==============================] - 2s 73ms/step - loss: 0.9257 - accuracy: 0.6975 - val_loss: 1.4244 - val_accuracy: 0.6000 # Now like the training data transformation to array I do the same with test data

# Convert the test dataframes into numpy arrays

mnist_test_array = np.array(test_df_org)

print("mnist_test_array.shape after converting to array ", mnist_test_array.shape)



# And also like the training data, need to do reshaping to get the structure which the model will expect to receive going forward

mnist_test_array = mnist_test_array.reshape(mnist_test_array.shape[0], 28, 28, 1)

print("mnist_test_array.shape after reshaping ", mnist_test_array.shape) mnist_test_array.shape after converting to array (1000, 784)

mnist_test_array.shape after reshaping (1000, 28, 28, 1)

Final Model Predictions and creating output file for Submission