Transfer Learning using VGG Pre-trained model with Keras | Towards Data Science

Transfer learning is one of the state-of-the-art techniques in machine learning that has been widely used in image classification. In this article, I will discuss transfer learning, the VGG model, and feature extraction. In the last section, I will demonstrate an interesting example of transfer learning where the transfer learning technique displays unexpectedly poor performance in classifying the MNist digit dataset.
VGG is a convolutional neural network with a specific architecture that was proposed in the paper — Very Deep Convolutional Networks for Large-Scale Image Recognition by a group of researchers (visual geometry group) from the University of Oxford. The VGG group participated in an annual computer vision competition — ImageNet Large Scale Visual Recognition Challenge (ILSVRC) and submitted the famous VGG model for competing in object localization (detecting objects within an image coming from 200 classes)and image classification tasks (1000-class classification). The ImageNet dataset, a major computer vision benchmark dataset that was used in the competition, includes more than 14 million images belonging to 1000 classes. The VGG model outperformed other models with 92.7% top-5 test accuracy and won 1st and 2nd place in the 2014 ILSVRC competition.
VGG Architecture
The VGG has two different architectures: VGG-16 contains 16 layers and VGG-19 contains 19 layers. In this article, we focus on VGG-16 which mainly contains three different parts: convolution, pooling, and fully connected layers — it starts with two convolution layers followed by pooling, then another two convolutions followed by pooling, after that repetition of three convolutions followed by pooling, and then finally three fully connected layers. The following figure shows the architecture of the VGG-16 model. The most interesting part of the VGG model is that the model weights are available on different platforms (i.e. Keras) and can be used for further analysis — developing models and applications. The idea of utilizing models’ weights for further tasks initiates the idea of transfer learning.
We can run the following three lines of code to plot the VGG-16 model architecture in python:
from keras.applications.vgg16 import VGG16from keras.utils import  plot_modelmodel = VGG16()plot_model(model)
Transfer Learning
We know that the training time increases exponentially with the neural network architecture increasing/deepening. In general, it could take hours/days to train a 3–5 layers neural network with a large-scale dataset. Consequently, deploying VGG from scratch on a large-scale dataset is a tiresome and computationally expensive task due to the depth and number of fully connected layers/nodes in the models’ architecture. Another challenge is that building VGG from scratch requires considerably large memory space and bandwidth since the size of ImageNet trained VGG-16 weights is 528 MB. However, instead of building a VGG from scratch, we can perform transfer learning i.e. — utilizing the knowledge like weights, and features of the previously trained (e.g. pre-trained VGG) models’ to solve a similar kind of problem. For instance, if we want to develop a binary image classifier, then we can use a pre-trained model that is trained on a large benchmark image dataset like VGG. Therefore, transfer learning is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task. Transfer learning is generally used for speeding up the training time and eventually improving the performance of the models’. While leveraging transfer learning, we must consider that the benefits of using transfer learning are not obvious.
When to Use Transfer Learning?
You can read the article Significance Of Transfer Learning In The World Of Deep Learning that explains when to use transfer learning.
Machine learning algorithms especially deep neural networks may not able to learn properly when the training data is not enough. Transfer learning can play a significant role to solve this issue and adjust the model to suit the new task. There are a few factors we can look for while applying transfer learning [1]:
Higher start: The initial stage of the model with transfer learning should outperform the model without transfer learning.Higher slope: The performance slope or rate of improvement during the training phase of the source model is steeper than it otherwise would be.Higher asymptote: The pre-trained should be converged more smoothly.
Higher start: The initial stage of the model with transfer learning should outperform the model without transfer learning.
Higher slope: The performance slope or rate of improvement during the training phase of the source model is steeper than it otherwise would be.
Higher asymptote: The pre-trained should be converged more smoothly.
Why the VGG model for transfer learning?
Assuming the given number of training samples (images) are not sufficient enough for building a classifier, in that case, VGG can be easily leveraged for feature extraction as it is trained on millions of images. VGG was introduced in 2014 and still, it mysteriously produces better accuracy for some specific tasks compared to other classifiers through many state-of-the-art Image classifiers that came after 2014.
In this article, we will show one demonstration of transfer learning with the VGG model that shows that the transfer learning approach does not perform up to the mark. In the future post, we will demonstrate another example where models’ performance improved as expected for leveraging the transfer learning method.
In this demonstration, we use the famous MNIST digit dataset. We use a portion of the MNIST dataset that contains only digits 8 and 6. Altogether there are 8200 samples and 784 features for each of the samples. We extract features for the MNIST dataset using the VGG pre-trained weights.
Step 1 — Data Preparation
We upload the dataset into the Google Colab from the local hard drive and convert the labels to 1 and 0 from 8 and 6.
from google.colab import filesuploaded = files.upload()
Now, we read the dataset. The following figure shows that the labels are 8 & 6. As in the later section, we will apply logistic regression, we convert the labels into 1 and 0.
Running the one line of code will convert the labels:
data['label'] = np.where(data['label'] == 8, 1, 0)
Step 2 — Extracting Features from the VGG-16 Model
Before extracting features, just observe the summary of the VGG-16 model:
In the summary, the VGG-16 contains 16 layers where the number of features is 25,088 after flatten of the last convolutional layer (the 1st highlighted) and in the final layer (prediction or final dense layer), the number of nodes is 1000 as VGG-16 mainly trained for 1000-class classification problem (the 2nd highlighted).
For transfer learning purposes, we can extract 25088 features from the MNIST dataset using the VGG model. The red vertical line shows the cut for feature extraction.
The following code section will cut the VGG model after the final convolutional layer:
from keras import  modelsbase_model = VGG16(weights='imagenet')model_VGG16 = models.Model(inputs=base_model.input, outputs=base_model.get_layer('flatten').output)
output:
After extracting features from the digit data using the VGG model, we trained a logistic regression binary classifier with the features and perform a 10-fold cross-validation. Simultaneously, we also apply logistic regression on the raw mnist digit data with 10-fold cross-validation to compare results with the performance of transfer learning.
The following two tables show the results of applying logistic regression to the raw data and extracted features from the raw data. Without transfer learning i.e. applying LR to the raw digit data achieves approximately 98% accuracy, while the model that leverages the transfer learning features achieves only 50% accuracy which is similar to a random guess for a binary classification problem. In general, we expect that with the transfer learning the model performance should be increased. However, in this demonstration, we experience that with transfer learning the model performance dramatically dropped.
The possible reasons that VGG16 does not perform well:
VGG-16 is trained for 3-channel RGB images while Mnist digit data is 1-channel grayscale
VGG-16 is trained for 3-channel RGB images while Mnist digit data is 1-channel grayscale
2. Background noises in the ImageNet data that were learned by the VGG-16 higher representational features
3. VGG-16 trained for 1000-class classification while for this task we used it for binary classification
Though the model with the transfer learning does not provide rewarding results in this experiment, utilizing other layers of the VGG for the feature extraction process or fine-tuning the parameters may produce better accuracy. To sum up, transfer learning is a state-of-the-art machine learning technique that can boost models’ performance. However, smartly utilizing the transfer learning technique is a requirement to gain the proper advantages.
Your membership fee will directly support and inspire Mohammad Masum and thousands of other writers you read. You’ll also get full access to every story on Medium — https://masum-math8065.medium.com/membership