Beginners Guide To Transfer Learning with an example using VGG16
By: Isha Puthige
Tags: Deep Learning, Transfer Learning, Convolution Neural Net, AI, Keras

When To Use Transfer Learning?

As stated in the chapter ‘Transfer Learning’ of the book ‘Handbook of Research on Machine Learning Applications and Trends’ we should consider using transfer learning if we reap any of the below stated benefits:

Higher Slope- The rate of performance improvement should be more than it would usually be. Higher Start-Performance initially achieved using only the transferred knowledge without refining or further learning is higher or better than that otherwise. Less Time Taken- The amount of time taken for the model to completely learn target class considering the transferred knowledge should be lesser than what a model made from scratch would take. Higher Asymptote- The final performance of the model with transfer learning should be greater than that without.

Performance Vs Training Graph

If the performance has actually decreased after applying transfer method, then negative transfer has occurred. This is one of the major challenges in applying transfer learning .We need to produce positive transfer between appropriately related tasks while avoiding negative transfer between tasks that are less related.

Notations and Definitions

Domain: domain D comprises of a marginal probability distribution P(X) and a feature space ꭕ. Mathematically domain can be represented as D = {ꭕ, P(Χ)}. Task: A task T comprises of a 2 element tuple of objective function, η and label space γ. From a probabilistic point of view the objective function can be denoted P(γ| Χ).

As stated in the paper ‘ A survey on Transfer Learning ’ by Sinno Pan and Qiang Yang the mathematical definition of transfer learning goes as follows:

‘Given a source domain DS and learning task TS, a target domain DT and learning task TT , transfer learning aims to help improve the learning of the target predictive function fT (·) in DT using the knowledge in DS and TS, where DS ≠ DT , or TS ≠TT ’

How To Use Transfer Learning?

We can either follow :

A Develop Model Approach: which consists of selecting a source task and developing a skillful model for it while also ensuring that some feature learning has been performed and then reusing that model on our second task of interest after performing some fine tuning

OR

A Pre- Trained Model Approach: which consists of selecting a pre trained source model out of the many models available which have been released by many famous institutions and research centers. We can reuse this model as our starting point on our 2nd task of interest. Most of the times we need to fine tune or adapt the model to apply it.

Some Examples

Using image data as input while performing predictive modelling is very common in transfer learning. There is also a wide use of it in solving NLP problems .

Models using Image Data

VGG Model — Developed by Oxford Inception Model — Developed by Google ResNet Model — Developed by Microsoft

Models Using Language Data

word2vec Model — Developed by Google GloVe Model — Developed by Stanford

Using them is really convenient because we can freeze all the layers(which means no need to train them again) and just train the lower layers making the process of retraining less cumbersome.

Implementing VGG Model to Classify An Object

VGG is an award winning model implemented by the Oxford Researchers.It is a convolutional neural network model developed as an improvement to the AlexNet.

ImageNet

ImageNet which is organized by the WordNet hierarchy is an image database consisting of around 14 million images with all the images having annotations done manually by humans. Since 2010 an image classification challenge called ILSVRC or the ImageNet Large Scale Visual Recognition Challenge is being conducted. The Oxford Visual Geometry Group won the award in 2014 by developing a convolutional neural network model.

Lets start off with the VGG-16 model

1) Load the VGG Model In Keras

An application interface is provided by keras for using and loading pre trained models.

2) Check Summary Of the Layers For Better Understanding

The base model of this particular model contains 16 layers. VGG16 is used for classification of images into various categories as it is pretrained on a large range of images providing rich feature representation. It accepts an input image of size (224,224,3). The initial layers comprises of convolution layers and convolution plus max-pooling layers this is followed by fully connected layers in the end.

Output would be:

3)Load The Image

I have used this image for this code demo.

First we load the image and then resize it to (224,224,3) to make it compatible with the model for input. In order to work with keras we also would have to convert the pixels to Numpy array. Generally the network is given multiple images as input, so it accepts an extra dimension(no. of samples) as a part of its input along with the rows, columns and channels. This brings the need to reshape our existing array. The image pixels need to adhere to the same format in the way the ImageNet training data was prepared (according to the research paper).Basically the mean RGB value calculated on the training set is subtracted from each pixel. Keras already provides with a function preprocess_input() for this purpose.

4) Prediction Time

model.predict() gives a list of length 1000 which consists of the probabilities of the image belonging to any of these known object labels. The keras function decode_predictions() provides us with the list of the topmost three object classes along with their probabilities. We just print the top most one.

Output:

fox_squirrel (96.02%)

Running the above code we get a prediction of the squirrel with 96% likelihood. Feel free to experiment and play with the above example since it will give you a better understanding.

Extending It To Transfer Learning

If you want to apply it for transfer learning you need to know about the arguments that VGG() takes.

include_top (True or False): It signifies whether or not the output layers are to be included. Generally when fitting to your own problem we do not include it. weights (‘imagenet’): Signifies the weights that you want to load. We generally specify the imagenet weights but if you want to train the model from scratch you can choose to write ‘None’. input_tensor (): Adding a new input layer incase you want fit the model on new data with a varied size. input_shape (): Incase you have changed the input layer you will have to specify the expected size of image the model needs to take in. pooling (): If you would be training a new set of output you need to mention the type of pooling layer (max pooling , average pooling etc) you would be using. classes ():output vector size of the model need to be specified within the brackets. That would signify the number of classes to be considered in the model. Generally we take 1000 .

Further reading