Beginner’s guide to transfer learning on Google Colab | by Suchandra Datta | Towards Data Science

Mammoth quantities of pristine data are one of the most valuable resources in these times, which is the potential source of huge revenue thanks to advances in deep learning and associated hardware that’s needed to speed up those innumerable matrix multiplications. What if we don’t have a lot of data and procuring more isn’t feasible? Or we lack the expensive hardware that’s imperative for training very deep networks? Both can be solved by using the concept of transfer learning, which we’ll soon find out is something we are unconsciously familiar with.
Transfer learning is a supervised learning method that aids construction of new models using pre-trained weights of previously constructed and fine-tuned models. Recall those situations in your life wherein you had to take a decision but lacked appropriate domain knowledge for the same. One course of action is to approach your family, professors, friends and peers; listen to their understanding of the problem, apply your own intuitions and finally arrive at an informed decision. This is exactly what happens in transfer learning too. Transfer learning can be applied in several different ways
Use the pre-trained models as it is: From our analogy, this is equivalent to listening to advice from others and applying it without modifications to the problem at hand.
As a feature extractor: We utilize the features the model has already learnt and expand the model further so that it can also effectively learn new features that are characteristic or peculiar to our dataset and may vary from the dataset on which the pre-trained model has been trained on. This is analogous to forming an opinion based on one’s own wisdom, coupled with advice from others.
Fine-tuning the model: In this case, we train the entire model, adding some layers of our own to get maximum performance. Whereas in feature extraction, we freeze the pre-trained layers to make sure learnt features aren’t updated during backpropagation, we unfreeze all layers in this approach as we hope this way the model generalizes better for the current dataset.
What do we achieve by transfer learning?
It’s one of the best options to choose when we need to build high performing models with small sized datasets with lack of time, money and resources needed to get a larger, unbiased dataset.
Lack of the huge amount of time needed to train very deep networks.
Lack of necessary hardware
In short, you don’t have to start from scratch however it’s highly recommended that you do train some feasibly small networks to understand the basics well before migrating to transfer learning.
Lack of expensive GPU’s can be compensated by using platforms like Google’s Colaboratory, a Google research project that makes machine learning accessible to all irrespective of hardware requirements, sporting a Jupyter Notebook environment, GPU or TPU support and all of this brought to you via cloud. To get more info on the specs, check out their example notebook linked here https://colab.research.google.com/drive/1_x67fw9y5aBW72a8aGePFLlkPvKLpnBl
We will apply this technique on a simple binary image classification problem, differentiation between cat and dog images. Our dataset is pretty simple, with total 1,000 images of dogs and cats, encoded with labels 0 and 1 respectively. For image classification, there are lots of established network architectures that we can go through to derive our model from scratch, prior to using pre-trained ones. LeNet-5 is the first one that was employed for handwritten digit classification. With the advent of ImageNet Large Scale Visual Recognition Challenge, AlexNet was born, along with others like VGG-16, VGG-19, ResNet50, Inception, Xception, MobileNet and the ultimate NASNet, which is a neural network model that figures out its own neural network architecture. A simple model based on AlexNet gets a training accuracy of 93% and testing accuracy of 83%. Theoretically, the minimum error that models for such type image classifications can make should be similar to human level error. Humans are adept at classifying dog and cat images so our model should exhibit an accuracy of 95% and more.
Let’s try out transfer learning using VGG-16
from keras.applications.vgg16 import VGG16from keras.models import Modelfrom keras.layers import Densefrom keras.layers import Flattenfrom keras.layers import Dropoutimport numpy as npfrom keras.preprocessing.image import ImageDataGeneratorfrom tensorflow.keras import regularizersvgg_model=VGG16(include_top=False, input_shape=(64,64,3))for layer in vgg_model.layers:        layer.trainable=False
Among the many pre-trained models available under keras.applications, we choose VGG-16 as it’s architecture is similar to the one we’ve already used. To use the model as a feature extractor, we set include_top=False, so we can add our custom ANN or CNN onto the existing model. Further to make sure that pre-trained weights aren’t adjusted during backpropagation, we set layer.trainable=False. Later on, however, it’s found that accuracy with this setting doesn’t rise above 78% so it will be commented out to perform fine-tuning.
x = vgg_model.output#add flatten layer so we can add the fully connected layer later#This is using the Keras functional API but Sequential API works #just as wellx = Flatten()(x)x = Dense(64, activation='relu')(x)x = Dropout(0.2)(x)x = Dense(1, activation='sigmoid')(x)#create the new modelmodel = Model(input=vgg_model.input, output=x)print(model.summary())
Next, the custom model, here a simple ANN is added on. Then the final model is constructed. Image data augmentation is applied to compensate for the size of the dataset. After some hyperparameter tuning, training and testing accuracy of 98% and 91% were obtained, for only 37 epochs, whereas we needed 70 epochs to get 83% accuracy on the one made from scratch. The same approach was also applied to ResNet50 which gave 92% and 87% as train and test accuracy. None of these however are perfect, with more tuning, more accuracy will be obtained.
On a closing note, some things that cropped up during transfer learning
All pre-trained models expect RGB color images.
All models have a minimum image size which can’t be changed arbitrarily, otherwise after a series of convolutions, the later layers will be expected to perform matrix multiplication of arrays of incompatible sizes.
Small batch sizes < 128 aren’t helpful for the current problem.
ResNet50, MobileNet and similar architectures have a BatchNormalization layer. During training, these layers aren’t updated, so the normalization is done with pre-computed values from the dataset on which this has been trained previously. If these values are widely different from the current dataset, low accuracy will be obtained. Unfreezing the layers yielded better results. (Frankly this is what I understood from this discussion here https://github.com/keras-team/keras/issues/9214#issuecomment-397916155 feel free to point out mistakes and add your viewpoints)
Dog and cat images were correctly identified after training as expected. It also started classifying lions as cats and wolves as dogs. It classifies cartoon images of dogs and cats too.
Here’s my humble attempt at demystifying some of the underlying concepts governing transfer learning and hopefully ease fellow coders’ transition into the land of deep learning. I’m pretty much a beginner myself, dabbling with these as I find it insanely interesting and cool, striving to be a self-taught data scientist. If you are in the same situation too, I hope you soon get the model accuracy that you’re struggling for right now. Thanks for reading this far, feel free to point out bugs, share your thoughts and happy learning!