Train a CNN using Skorch for MNIST digit recognition
By: Eugenia Anello
Tags: Mnist, Skorch, Artificial Intelligence, Convolutional Network, Data Science

Train a CNN using Skorch for MNIST digit recognition

Image made by Author

The goal of this post is to use a tool to train and evaluate a Pytorch’s model in a simple way. This tool is Skorch, that is a scikit-learn compatible neural network library that wraps Pytorch. So it makes possible to use Pytorch with sklearn. Moreover, it takes advantage of Scikit-learn’s functions such as fit, predict and GridSearch [1]. This tool is applied on MNIST, a dataset composed by images of handwritten digits: 60,000 for training and 10,000 for testing. In the convolutional neural network, we’ll take as input the image of a handwritten digit (0–9) and classify the image as the appropriate digit.

What is Skorch?

Skorch is a very recent open-source library, launched the first time in 2017. In a nutshell, it combines SciKit-Learn and pythORCH. Thus, it allows to easily swap neural networks with SKlearn and extract Pytorch modules. The main advantage of this open-source is the reduction of the boilerplate code. If you have used Pytorch in the past, writing the for training can be problematic, there is always the same code to write and it’s very easy to do mistakes having many lines [2].

Training with Pytorch:

Training with Skorch

Comparing the two examples, you can observe Skorch allows to use less lines of code. In order to make understand this open-source library works, I will walk through the classic MNIST classification problem.

Step 1: Import libraries and dataset

Let’s install Skorch, the library we’ll be using in all this tutorial:

Once the library is installed, we can import the libraries.

skorch.NeuralNetClassifier is a Neural Network class used for classification tasks.

Step 2: Prepare the dataset before training

We are creating an object using torch.device. The Tensor will be allocated in this device object, whose device type is cpu or cuda. To use the gpu, you need to specify “cuda”.

If we want skorch to do a validation split for us, we need to retrieve the y_train values from the training dataset and pass these values to net.fit later on:

Step 3: Create the model

Our convolution neural network has two convolution layers, each with 5x5 kernel, and two fully connected layers.

Step 4: Train Model

Now we are going to initialize the NeuralNetClassifier class, in which we specify:

Pytorch Module : in our case our CNN model.

: in our case our CNN model. criterion (default=nn.NLLLoss): we need to specify that we are using CrossEntropyLoss instead.

(default=nn.NLLLoss): we need to specify that we are using CrossEntropyLoss instead. optimizer (default=optim.SGD): we select the optimizer Adam.

(default=optim.SGD): we select the optimizer Adam. batch_size (default=128): we use 64 as batch_size.

(default=128): we use 64 as batch_size. lr (default=0.01): the learning rate passed to the optimizer is 1e-3.

(default=0.01): the learning rate passed to the optimizer is 1e-3. max_epochs (default=10)

(default=10) iterator_train and iterator_valid: The default PyTorch DataLoader used for training and validation data.

and The default PyTorch used for training and validation data. train_split(default=0.2): By default, 20% of the training data is reserved for validation.

Thus, the training data will be split internally by skorch into 80% training and 20% validation by default.

After the initialization, we will train the CNN model using the method fit:

The results show the training loss, the validation loss, the validation accuracy and the duration of each epoch. The model seems to work well since there is a slight difference between the two losses. It’s also evident looking at the plot below, obtained through the attribute history of the model “net”. Basically, it’s a list of dicts that contains the information about the model training: for each epoch there is an element, that, again, contains a list of dicts for each batch.

Step 5: Prediction

We’ll predict on the test data and evaluate the accuracy of the prediction using the functions predict and accuracy_score from sklearn library.

An accuracy of 99% is very good for a simple network. Another measure to evaluate the performance of the model is the confusion matrix, which compares the actual classes with the predicted classes. In diagonal, there is the number of points for which the predicted label is equal to the true label, while in the off-diagonal there are the misclassified elements.

Even if the overall accuracy is 99%, there are still many misclassified data. In particular, the class 4 is classified wrongly as 9 nine times. For this reason, we’ll try K-Fold Cross Validation to be sure to avoid overfitting and not loose generalization.

Step 6: Evaluate the score by K-fold Cross Validation

Before the evaluation, we need to make the training set sliceable using the class SliceDataset, that wraps a torch dataset to make it work with sklearn. We’ll use the function cross_val_score (estimator, X, y=None, cv=None) of sklearn library to evaluate the validation accuracy obtained using K=5 folds for K-Fold Cross Validation.

The results are related to the first two folds, but you also should see the other three tables like these.

Below there are the average accuracy of each fold and the average accuracy of all the folds:

Even if it’s computationally expensive, K-fold cross-validation is needed to evaluate the model without overfitting. The accuracy remains pretty high, so it means the model works well with any resampling procedures.