Classifying Handwritten Digits Using A Multilayer Perceptron Classifier (MLP) | by Serafeim Loukas | Towards Data Science

An MLP is a supervised machine learning (ML) algorithm that belongs in the class of feedforward artificial neural networks [1]. The algorithm essentially is trained on the data in order to learn a function. Given a set of features and a target variable (e.g. labels) it learns a non-linear function for either classification or regression. In this article, we will only focus on the classification case.
There is! The logistic regression has only two layers i.e. the input and output however, in the case of a MLP model, the only difference is that we can have additional intermediate nonlinear layers. These are known as hidden layers. Except for the input nodes (nodes that belong to the input layer), each node is a neuron that uses a nonlinear activation function [1]. Due to this non-linearity nature, the MLP can learn complex nonlinear functions and thus, distinguish data that are not linearly separable! See Figure 2 below for a visual representation of an MLP classifier with one hidden layer.
MLP uses backpropagation for training [1]. You can have a look at this website here for the formal mathematical formulation.
Pros
Can learn nonlinear functions and thus separate not linearly separable data [2].
Cons
The loss function for the hidden layer leads to a non-convex optimization problem and thus, local minimum exist.
Due to the above issue, different weight initializations may lead to different outputs/weights/results.
The MLP has some hyperparameters e.g. the number of hidden neurons, layers that need to be tuned (time & power consuming) [2].
MLP can be sensitive to feature scaling [2].
For this hands-on example we will use the MNIST dataset. The MNIST database is famous database of handwritten digits that is used for training several ML models [5]. There are handwritten images of 10 different digits so the number of classes is 10(see Figure 3).
Note: Since we deal with images, these are represented by 2D arrays and the initial dimension of the data is 28 by 28 for each image (28x28 pixels). The 2D images are then flattened and thus, represented by vectors at the end. Each 2D image is transformed into a 1D vector with dimension [1, 28x28] = [1, 784]. Finally, our dataset has 784 features/variables/columns.
import matplotlib.pyplot as pltfrom sklearn.datasets import fetch_openmlfrom sklearn.neural_network import MLPClassifier# Load dataX, y = fetch_openml("mnist_784", version=1, return_X_y=True)# Normalize intensity of images to make it in the range [0,1] since 255 is the max (white).X = X / 255.0
Remember that each 2D image is now transformed into a 1D vector with dimension [1, 28x28] = [1, 784]. Let’s verify this now.
print(X.shape)
This returns: (70000, 784). We have 70k flattend images (samples), each containing 784 pixels (28*28=784) (variables/features). Thus, the input layer weight matrix will have shape 784 x #neurons_in_1st_hidden_layer. The output layer weight matrix will have shape #neurons_in_3rd_hidden_layer x #number_of_classes.
Now let’s build the model, train it and perform the classification. We will use 3 hidden layers with 50,20 and 10 neurons each, respectively. Also, we will set the max iterations to 100 and the learning rate to 0.1. These are the hyperparameters that I mentioned in the Introduction. We will not fine-tune them here.
# Split the data into train/test setsX_train, X_test = X[:60000], X[60000:]y_train, y_test = y[:60000], y[60000:]classifier = MLPClassifier(    hidden_layer_sizes=(50,20,10),    max_iter=100,    alpha=1e-4,    solver="sgd",    verbose=10,    random_state=1,    learning_rate_init=0.1,)# fit the model on the training dataclassifier.fit(X_train, y_train)
Now, let’s evaluate the model. We will estimate the mean accuracy on the training and test data and labels.
print("Training set score: %f" % classifier.score(X_train, y_train))print("Test set score: %f" % classifier.score(X_test, y_test))
Training set score: 0.998633Test set score: 0.970300
Great results!
How fast was the loss decreased during training? Let’s make a nice plot!
fig, axes = plt.subplots(1, 1)axes.plot(classifier.loss_curve_, 'o-')axes.set_xlabel("number of iteration")axes.set_ylabel("loss")plt.show()
Here we see that the loss decreases really fast during training and it saturates after the 40th iteration (remember that we defined max 100 iterations as hyperparameter).
Here we need first to understand how the weights (learnt model parameters of each layer) are stored.
Based on the documentation, the attribute classifier.coefs_ is a list of shape (n_layers-1, )of weight arrays, where weight matrix at index i represents the weights between layer i and layer i+1. In this example, we defined 3 hidden layers and we also have the input and output layers. Thus, we expect to have 4 weight arrays for the between-layers weights (in-L1, L1-L2, L2-L3 and L2-out in Figure 5).
Similarly, classifier.intercepts_is a list of bias vectors, where the vector at index i represents the bias values added to layer i+1.
Let’s verify this:
len(classifier.intercepts_) == len(classifier.coefs_) == 4
correctly returns True.
Reminder: The input layer weight matrix will have shape 784 x #neurons_in_1st_hidden_layer. The output layer weight matrix will have shape #neurons_in_3rd_hidden_layer x #number_of_classes.
target_layer = 0 #0 is input, 1 is 1st hidden etcfig, axes = plt.subplots(1, 1, figsize=(15,6))axes.imshow(np.transpose(classifier.coefs_[target_layer]), cmap=plt.get_cmap("gray"), aspect="auto")axes.set_xlabel(f"number of neurons in {target_layer}")axes.set_ylabel("neurons in output layer")plt.show()
Or reshape and plot them as 2D images again.
# choose layer to plottarget_layer = 0 #0 is input, 1 is 1st hidden etcfig, axes = plt.subplots(4, 4)vmin, vmax = classifier.coefs_[0].min(), classifier.coefs_[target_layer].max()for coef, ax in zip(classifier.coefs_[0].T, axes.ravel()):    ax.matshow(coef.reshape(28, 28), cmap=plt.cm.gray, vmin=0.5 * vmin, vmax=0.5 * vmax)    ax.set_xticks(())    ax.set_yticks(())plt.show()
MLP classifier is a very powerful neural network model that enables the learning of non-linear functions for complex data. The method uses forward propagation to build the weights and then it computes the loss. Next, back propagation is used to update the weights so that the loss is reduced. This is done in an iterative way and the number of iterations is an input hyperparameter as I explained in the Introduction. Other important hyperparameters are the number of neurons in each hidden layer and the number of hidden layers in total. These need to be fine-tuned.
That’s all folks! Hope you liked this article!
If you liked and found this article useful, follow me!
Questions? Post them as a comment and I will reply as soon as possible.
medium.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
[1] https://en.wikipedia.org/wiki/Multilayer_perceptron
[2] https://scikit-learn.org/stable/modules/neural_networks_supervised.html#mlp-tips
[3] https://scikit-learn.org/stable/modules/neural_networks_supervised.html#mathematical-formulation
[4] https://en.wikipedia.org/wiki/Backpropagation
[5] https://en.wikipedia.org/wiki/MNIST_database
Mailing list: https://seralouk.medium.com/subscribe