Transfer Learning using a Pre-trained Model
By: Sharon Lim
Tags: Pokemon, Transfer Learning, Image Classification, Using Pretrained Model, Vgg16

Transfer Learning using a Pre-trained Model

Exploring the use of VGG16 for Image Classification

What is Transfer Learning?

Transfer learning is a research problem in machine learning that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem.

Traditional Machine Learning VS Transfer Learning (source: Dipanjan Sarkar)

The traditional machine learning approach generalizes unseen data based on patterns learned from the training data, whereas for transfer learning, it begins from previously learned patterns to solve a different task.

Basic Idea of Transfer Learning (source: Integrate.ai)

There are two common approaches to transfer learning:

Developed model approach: Develop a model that is better than a naive model to ensure that some feature learning has been performed. Reuse and tune the developed model for the task of interest. Pre-trained model approach: Select a pre-trained source model from available models to reuse and fine-tune.

In this post, we shall focus on the pre-trained model approach as it is commonly used in the field of deep learning.

What are Pre-trained Models?

A pre-trained model is a saved network that was previously trained on a large dataset, typically on a large-scale image-classification task. One can use the pre-trained model as it is or use transfer learning to customize this model to a given task.

The intuition behind transfer learning is that if a model is trained on a large and general enough dataset, this model will effectively serve as a generic model of the visual world. We can then take advantage of these learned feature maps without having to start from scratch by training a large model on a large dataset.

Let’s take a deep dive into VGG16 - a notable pre-trained model submitted to the Large Scale Visual Recognition Challenge in 2014.

VGG16 Architecture

The VGG network architecture was introduced by Simonyan and Zisserman in their 2014 paper, Very Deep Convolutional Networks for Large Scale Image Recognition. This model had achieved a 92.7% top-5 test accuracy in ImageNet, which is a dataset of over 14 million images belonging to 1,000 classes. The number ‘16’ in VGG16 refers to its 16 layers which have weights. The network is considerably large with approximately 138 million parameters.

Architectures of VGG16 (source: Neurohive)

The input to conv1 layer is of fixed size 224 x 224 RGB image. The image is passed through a stack of convolutional layers, where the filters were used with a very small receptive field: 3×3 (which is the smallest size to capture the notion of left/ right, up/ down, center). In one of the configurations, it also utilizes 1×1 convolution filters, which can be seen as a linear transformation of the input channels (followed by non-linearity).

The convolution stride is fixed to 1 pixel; the spatial padding of convolution layer input is such that the spatial resolution is preserved after convolution, i.e. the padding is 1-pixel for 3×3 convolution layers. Spatial pooling is carried out by five max-pooling layers, which follow some of the convolution layers (not all the convolution layers are followed by max-pooling). Max-pooling is performed over a 2×2 pixel window, with stride 2.

Three fully connected layers follow a stack of convolutional layers (which has a different depth in different architectures): the first two have 4,096 channels each, the third performs 1,000-way ILSVRC classification and thus contains 1,000 channels (one for each class). The final layer is the soft-max layer. The configuration of the fully connected layers is the same in all networks.

All hidden layers are equipped with the rectification (ReLU) non-linearity. It is also noted that none of the networks (except for one) contain Local Response Normalisation (LRN), such normalization does not improve the performance on the dataset, but leads to increased memory consumption and computation time.

We shall approach a problem statement and explore the basics of how transfer learning using a pre-trained model works, how to fine-tune the hyper-parameters to improve model performance, and interpret the learning curves.

Problem Statement: Using Transfer Learning, build a Convolutional Neural Network (“CNN”) model to classify Pokémons into their respective categories.

Let’s code and catch ‘em all!

Data Preparation

I have selected 5 categories of Pokémons from the Pokémons Dataset. Load these images and plot a bar chart to get a feel of how many images there are in each class.

With the above bar chart, there is a good average size of at least 280 images per Pokémon class. Due to hardware constraints, we would work with an input image dimension of 128 X 128 instead of the original 224 X 224 used in the VGG16 model. We shall proceed to read, resize, and scale these images for training.

A total of 1,429 images with input dimensions of 128 X 128 X 3

Load and split the dataset for training and testing of the CNN model. A test size of 0.2 was used, meaning 80% of the dataset would be used for training purposes while the remaining 20% serves as the testing component. We then split the dataset in a stratified fashion, using the labels, y, as the class labels. The dataset is also shuffled before splitting, to enhance the randomness of probability sampling. The random state controls the shuffling applied to the data before applying the split. An integer is passed in this parameter for reproducible output across multiple function calls.

Creation of CNN Model using Transfer Learning

Transfer learning is applied by freezing the “deep layers” of the model and only re-training the classification layers.

We have selected to use the Adam optimization, which is a stochastic gradient descent method that is based on an adaptive estimation of first-order and second-order moments.

According to Kingma et al., 2014, this method is

“computationally efficient, has little memory requirement, invariant to diagonal re-scaling of gradients, and is well suited for problems that are large in terms of data/ parameters.”

Model Training