CNN Transfer Learning & Fine Tuning | by Victor Roman | Towards Data Science

Learn how to apply these powerful techniques to take your Deep Learning models to a whole new level!
As we have seen in previous articles, we can use the architectures developed by research teams and leverage their power to make predictions and obtain better results in our Deep Learning models.
Training a neural network takes time, luckily nowadays there are ways to avoid having to:
Define the architecture of a neural network
Train her from the beginning
We already have seen ways to avoid having to define the architecture here, and it consists of using predefined architectures that are known to work well: ResNet, AlexNet, VGG, Inception, DenseNet, etc.
And what about avoiding training it from scratch? What do I mean by that?
Neural networks are initialized with random weights (usually) that after a series of epochs reach some values that allow us to properly classify our input images.
What would happen if we could initialize those weights to certain values that we know beforehand that are already good to classify a certain dataset?
In this way, we would not need a dataset as big as if we were to train a network from zero (from hundreds of thousands or even millions of images we could go to a few thousands) nor would we need to wait a good number of epochs for the weights to take good values for the classification, they would have it much easier due to their initialization.
Let’s explore how to do this with the techniques of transfer learning and fine-tuning:
Let’s take the VGG16 network trained on the ImageNet dataset as an example. Let’s see its architecture:
We know that ImageNet consists of a dataset of about 1.2 million images for training, 50,000 for validation and 100,000 for testing, belonging to 1000 categories.
Now imagine that what we want is to apply the VGG16 trained on ImageNet to another dataset, let’s say that we choose the CIFAR-10. How could we do it?
Remembering the general scheme of a CNN, we have a feature extractor in the first stage and then a classifier:
What if we removed the last layer of the VGG16, which simply takes a probability for each of the 1000 classes in the ImageNet and replaces it with a layer that takes 10 probabilities? This way, we could take all the knowledge that VGG16 has trained on the ImageNet and apply it to our problem!
As we have seen, what we will do is change the classification stage, so that the last layer is one of 10 neurons (our CIFAR 10 has 10 classes) and then we will retrain the network allowing the weights of the fully connected layers to be changed, that is, the classification stage.
For this, we would initialize our network with the weights from the ImageNet, and then freeze all the convolutional and max-pooling layers so that they do not modify their weights, leaving only the fully connected ones free.
Once this is done, we would start retraining. In this way, we manage to take advantage of the feature extraction stage of our network and only tune the final classifier to work better with our dataset. This is what is known as Transfer Learning because we take advantage of the knowledge of another problem to solve the one we are dealing with.
This approach can also be done by saving the characteristics given by the last layer of max pooling and then putting that data into any classifier (SVM, logreg, etc).
Let’s see how we could do it:
# We first load the necessary libraries, the dataset and reshape its dimensons to the minimum allowed by the VGG16 --> (48,48,3)import tensorflow as tffrom keras import callbacksfrom keras import optimizersfrom keras.engine import Modelfrom keras.layers import Dropout, Flatten, Densefrom keras.optimizers import Adamfrom keras.applications import VGG16from keras.datasets import cifar10from keras.utils import to_categoricalimport numpy as npinput_shape = (48, 48, 3)(X_train, y_train), (X_test, y_test) = cifar10.load_data()Y_train = to_categorical(y_train)Y_test = to_categorical(y_test)# resize train setX_train_resized = []for img in X_train:  X_train_resized.append(np.resize(img, input_shape) / 255)  X_train_resized = np.array(X_train_resized)print(X_train_resized.shape)# resize test setX_test_resized = []for img in X_test:  X_test_resized.append(np.resize(img, input_shape) / 255)  X_test_resized = np.array(X_test_resized)print(X_test_resized.shape)
# We build the base modelbase_model = VGG16(weights='imagenet', include_top=False, input_shape=input_shape)base_model.summary()
# We freeze every layer in our base model so that they do not train, we want that our feature extractor stays as before --> transfer learningfor layer in base_model.layers:   layer.trainable = False  print('Layer ' + layer.name + ' frozen.')# We take the last layer of our the model and add it to our classifierlast = base_model.layers[-1].outputx = Flatten()(last)x = Dense(1000, activation='relu', name='fc1')(x)x = Dropout(0.3)(x)x = Dense(10, activation='softmax', name='predictions')(x)model = Model(base_model.input, x)# We compile the modelmodel.compile(optimizer=Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])-model.summary()
# We start the trainingepochs = 10batch_size = 256# We train itmodel.fit(X_train_resized, Y_train,          batch_size=batch_size,          validation_data=(X_test_resized, Y_test),          epochs=epochs)
# We evaluate the accuracy and the loss in the test setscores = model.evaluate(X_test_resized, Y_test, verbose=1)print('Test loss:', scores[0])print('Test accuracy:', scores[1])
We haven’t had to train at all, and we’ve had not bad results! Keep in mind that if we did it randomly the probability of getting it right would be 1/10=0.1 or 10%, since we have 10 classes.
The more similar the dataset on which the network was originally trained and the dataset of our problem are, the better results we get.
And if our dataset has nothing to do with the one of ImageNet or we want to improve the results even more?
Well, for that, we use fine-tuning.
With fine-tuning, we first change the last layer to match the classes in our dataset, as we have done before with transfer learning. But besides, we also retrain the layers of the network that we want.
Remember again the architecture of the VGG16:
What we did in the previous example was to change only the layers of the classification stage, keeping the knowledge that the network obtained when extracting characteristics (patterns) in the previous task, from which we are loading the weights (ImageNet).
With fine-tuning we are not limited to retraining only the classifier stage (i.e. the fully connected layers), but what we will do is retrain also the feature extraction stage, i.e. the convolutional and pooling layers.
It’s important to keep in mind that in a neural network, the first layers detect simpler and more general patterns, and the more we advance in the architecture, the more specific to the dataset and the more complicated the patterns they detect.
Therefore, we could allow the last block of convolution and pooling layers to be retrained.
When do I do fine-tuning and transfer learning? How do I choose which layer to retrain from?
Well, as a general rule, the first thing we will do is transfer learning, i.e. we will not retrain our network. That will give us a baseline that we will have to overcome. Then, we will retrain only the classification stage, and then we can try to retrain some convolutional block as well.
Make transfer learning, that is, modify only the last layer so that it has the same number of outputs as our classes (baseline)
Try to retrain the sorting stage, i.e. the dense layers
Trying to retrain some convolutional stage
Following these steps most of the time you will reach a suitable result for your problem
It also depends on the type of problem you have. If:
The new dataset is small and similar to the original one: be careful when fine-tuning, maybe it is better to choose the characteristics of the last layer of the convolutional stage and use an SVM or linear classifier.
The new dataset is large and similar to the original: having more data we probably won’t over-fit, so we can fine-tune with more confidence.
The new dataset is small and very different from the original: it would be best to use features of an earlier layer of the convolutional stage, as this will be set in more general patterns than the later layers, and then use a linear classifier.
The new dataset is big and very different from the original: we will train it from scratch! However, it is still recommended that you initialize the weights with those of the ImageNet.
When using any of these techniques, you must take into account the possible restrictions of the pre-trained models. For example, they may require a minimum image size.
Besides, when retraining networks, we usually choose lower learning rates than if we do it from scratch since we start from the initialization of weights that is assumed to be good.
# Fine Tuning Example, classification VGG16 with CIFAR 10, we import the necessary librariesimport tensorflow as tffrom keras import callbacksfrom keras import optimizersfrom keras.engine import Modelfrom keras.layers import Dropout, Flatten, Densefrom keras.optimizers import Adamfrom keras.preprocessing.image import ImageDataGeneratorfrom keras.applications import VGG16from keras.datasets import cifar10from keras.utils import to_categoricalimport numpy as np# We first load the dataset and reshape its dimensions to the minimum allowed by VGG16 --> (48, 48, 3)input_shape = (48, 48, 3)(X_train, y_train), (X_test, y_test) = cifar10.load_data()Y_train = to_categorical(y_train)Y_test = to_categorical(y_test)# resize train setX_train_resized = []for img in X_train:  X_train_resized.append(np.resize(img, input_shape) / 255)  X_train_resized = np.array(X_train_resized)print(X_train_resized.shape)# resize test setX_test_resized = []for img in X_test:  X_test_resized.append(np.resize(img, input_shape) / 255)  X_test_resized = np.array(X_test_resized)print(X_test_resized.shape)
# We build the base modelbase_model = VGG16(weights='imagenet', include_top=False, input_shape=input_shape)base_model.summary()
# We allow to the last convolutional andthe classification stages  to trainfor layer in base_model.layers:  if layer.name == 'block5_conv1':    break  layer.trainable = False  print('Layer ' + layer.name + ' frozen.')# We add our classificator (top_model) to the last layer of the modellast = base_model.layers[-1].outputx = Flatten()(last)x = Dense(1000, activation='relu', name='fc1')(x)x = Dropout(0.3)(x)x = Dense(10, activation='softmax', name='predictions')(x)model = Model(base_model.input, x)# We compile the modelmodel.compile(optimizer=Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])# We see the new structure of the modelmodel.summary()
# We start the trainingepochs = 15batch_size = 256# We train the modelmodel.fit(X_train_resized, Y_train,          batch_size=batch_size,          validation_data=(X_test_resized, Y_test),          epochs=epochs)
# We evaluate the accuracy and the loss in the test setscores = model.evaluate(X_test_resized, Y_test, verbose=1)print('Test loss:', scores[0])print('Test accuracy:', scores[1])
So,the way to go when we face a Deep Learning problem would be to star always using fine-tunning and building a baseline model which we will try to improve later.
As always, I hope you enjoyed the post, and that you gained an intuition about how to implement and develop a convolutional neural network with transfer learning and fine-tuning!
If you liked this post then you can take a look at my other posts on Data Science and Machine Learning here.
If you want to learn more about Machine Learning, Data Science and Artificial Intelligence follow me on Medium, and stay tuned for my next posts!