Deep Learning using Transfer Learning -Python Code for ResNet50 | by Renu Khandelwal | Towards Data Science

This is the second part of the series where we will write code to apply Transfer Learning using ResNet50 . Here we will use transfer learning suing a Pre-trained ResNet50 model and then fine-tune ResNet50.
Transfer Learning Concept part 1
For code implementation, we will use ResNet50. ResNet is short for Residual Network. It is a 50 layer Residual Network
When we add more layers to our deep neural networks, the performance becomes stagnant or starts to degrade. This happens due to vanishing gradient problem. When gradients are backpropagated through the deep neural network and repeatedly multiplied, this makes gradients extremely small causing vanishing gradient problem.
ResNet solves the vanishing gradient problem by using Identity shortcut connection or skip connections that skip one or more layers. Shortcut connections are connecting output on layer N to the input of layer N+Z
We will use Cats and Dogs data set for demonstrating Transfer Learning using
Pre-trained ResNet50 model as a feature extractor
Fine-tuning the pre-trained model ResNet50
Importing the basic libraries. We will import additional libraries when required
import globimport numpy as npimport pandas as pdimport osimport shutil import matplotlib.pyplot as pltfrom keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array, array_to_img%matplotlib inline
I have stored all the images of cats and dogs in the folder dogs-vs-cats folder. We read the cat and dog files. We have 25003 images each of cats and dogs
files = glob.glob('E:\\Data\\Images\\dogs-vs-cats\\*')  cat_files = [fn for fn in files if 'cat' in fn] dog_files = [fn for fn in files if 'dog' in fn] len(cat_files), len(dog_files)
I am training on a smaller set of data so reducing the size of my training, test and validation data set. This step is not required if you want to train on all the images.
Train data set will have 1500 images each of cats and dogs, Test data set will have 500 images each of cats and dogs and Validation data set will also have 500 images each of cats and dogs
cat_train = np.random.choice(cat_files, size=1500, replace=False) dog_train = np.random.choice(dog_files, size=1500, replace=False) cat_files = list(set(cat_files) — set(cat_train)) dog_files = list(set(dog_files) — set(dog_train))  cat_val = np.random.choice(cat_files, size=500, replace=False) dog_val = np.random.choice(dog_files, size=500, replace=False) cat_files = list(set(cat_files) — set(cat_val)) dog_files = list(set(dog_files) — set(dog_val))  cat_test = np.random.choice(cat_files, size=500, replace=False) dog_test = np.random.choice(dog_files, size=500, replace=False)  print(‘Cat datasets:’, cat_train.shape, cat_val.shape, cat_test.shape) print(‘Dog datasets:’, dog_train.shape, dog_val.shape, dog_test.shape)
Loading the training and validation data set. The dimension of our image will be 300 by 300 pixel
IMG_WIDTH=300IMG_HEIGHT=300IMG_DIM = (IMG_WIDTH, IMG_HEIGHT)train_files = glob.glob(‘E:\\Data\\Images\\dogs-vs-cats\\training_data\\*’)train_imgs = [img_to_array(load_img(img, target_size=IMG_DIM)) for img in train_files]train_imgs = np.array(train_imgs)train_labels = [fn.split(‘\\’)[-1].split(‘.’)[0].strip() for fn in train_files]validation_files = glob.glob(‘E:\\Data\\Images\\dogs-vs-cats\\validation_data\\*’)validation_imgs = [img_to_array(load_img(img, target_size=IMG_DIM)) for img in validation_files]validation_imgs = np.array(validation_imgs)validation_labels = [fn.split(‘\\’)[-1].split(‘.’)[0].strip() for fn in validation_files]print(‘Train dataset shape:’, train_imgs.shape,  ‘\tValidation dataset shape:’, validation_imgs.shape)
Each image is now of size 300 x 300 and has three channels for Red, Green, and Blue (RGB).
Pixel values for images are between 0 and 255. Deep Neural networks work well with smaller input values. Scaling each image with values between 0 and 1.
train_imgs_scaled = train_imgs.astype(‘float32’) validation_imgs_scaled = validation_imgs.astype(‘float32’) train_imgs_scaled /= 255 validation_imgs_scaled /= 255  # visualize a sample image print(train_imgs[0].shape) array_to_img(train_imgs[0]
Encoding text category labels of Cats and Dogs
# encode text category labels from sklearn.preprocessing import LabelEncoder  le = LabelEncoder() le.fit(train_labels) train_labels_enc = le.transform(train_labels) validation_labels_enc = le.transform(validation_labels)  print(train_labels[1495:1505], train_labels_enc[1495:1505])
Keras framework has an excellent utility called ImageDataGenerator. It generates batches of tensor image data with real-time data augmentation.
For our training and validation datasets, we will zoom the image randomly by a factor of 0.3 using the zoom_range parameter. We rotate the image randomly by 50 degrees using the rotation_range parameter. Translating the image randomly horizontally or vertically by a 0.2 factor of the image’s width or height using the width_shift_range and the height_shift_range parameters. Applying shear-based transformations randomly using the shear_range parameter. Randomly flipping half of the images horizontally using the horizontal_flip parameter. Leveraging the fill_mode parameter to fill in new pixels for images after we apply any of the preceding operations (especially rotation or translation). In this case, we just fill in the new pixels with their nearest surrounding pixel values.
train_datagen = ImageDataGenerator(rescale=1./255, zoom_range=0.3, rotation_range=50, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2,  horizontal_flip=True, fill_mode=’nearest’)val_datagen = ImageDataGenerator(rescale=1./255)
Let’s see how some of the augmented images looks like. We will take two sample images from our training dataset to illustrate the same. The first image is an image of a cat and the second image is of a dog
img_id = 2500cat_generator = train_datagen.flow(train_imgs[img_id:img_id+1],  train_labels[img_id:img_id+1],  batch_size=1) cat = [next(cat_generator) for i in range(0,5)] fig, ax = plt.subplots(1,5, figsize=(16, 6))print(‘Labels:’, [item[1][0] for item in cat]) l = [ax[i].imshow(cat[i][0][0]) for i in range(0,5)]
img_id = 4001 dog_generator = train_datagen.flow(train_imgs[img_id:img_id+1],  train_labels[img_id:img_id+1],  batch_size=1) dog = [next(dog_generator) for i in range(0,5)] fig, ax = plt.subplots(1,5, figsize=(15, 6)) print(‘Labels:’, [item[1][0] for item in dog]) l = [ax[i].imshow(dog[i][0][0]) for i in range(0,5)]
For our test generator, we need to send the original test images to the model for evaluation. We just scale the image pixels between 0 and 1 and do not apply any transformations.
We just apply image augmentation transformations only to our training set images and validation images
train_generator = train_datagen.flow(train_imgs, train_labels_enc,batch_size=30)val_generator = val_datagen.flow(validation_imgs, validation_labels_enc, batch_size=30)
We use ResNet50 deep learning model as the pre-trained model for feature extraction for Transfer Learning.
To implement Transfer learning, we will remove the last predicting layer of the pre-trained ResNet50 model and replace them with our own predicting layers. FC-T1 and FC_T2 as shown below
Weights of ResNet50 pre-trained model is used as feature extractor
Weights of the pre-trained model are frozen and are not updated during the training
We do not want to load the last fully connected layers which act as the classifier. We accomplish that by using “include_top=False”. We do this so that we can add our own fully connected layers on top of the ResNet50 model for our task-specific classification.
We freeze the weights of the model by setting trainable as “False”. This stops any updates to the pre-trained weights during training We do not want to train ResNet layers as we want to leverage the knowledge learned by the deep neural network trained from the previous data set which in our case is “imagenet”
from keras.applications.resnet50 import ResNet50from keras.models import Modelimport kerasrestnet = ResNet50(include_top=False, weights='imagenet', input_shape=(IMG_HEIGHT,IMG_WIDTH,3))output = restnet.layers[-1].outputoutput = keras.layers.Flatten()(output)restnet = Model(restnet.input, output=output)for layer in restnet.layers:    layer.trainable = Falserestnet.summary()
We now create our model using Transfer Learning using Pre-trained ResNet50 by adding our own fully connected layer and the final classifier using sigmoid activation function.
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, InputLayerfrom keras.models import Sequentialfrom keras import optimizersmodel = Sequential()model.add(restnet)model.add(Dense(512, activation='relu', input_dim=input_shape))model.add(Dropout(0.3))model.add(Dense(512, activation='relu'))model.add(Dropout(0.3))model.add(Dense(1, activation='sigmoid'))model.compile(loss='binary_crossentropy',              optimizer=optimizers.RMSprop(lr=2e-5),              metrics=['accuracy'])model.summary()
We see that the weights of ResNet50 are not trainable as we had frozen them.
we now run the model
history = model.fit_generator(train_generator,                               steps_per_epoch=100,                               epochs=100,                              validation_data=val_generator,                               validation_steps=50,                               verbose=1)
Saving the trained weights
model.save(‘cats_dogs_tlearn_img_aug_cnn_restnet50.h5’)
We can use deep neural networks like VGG-16, VGG-19, Inception V3, ResNet-50, Xception as pre-trained model
To implement Transfer learning with fine-tuning, we remove the last predicting layer of the pre-trained model and replace them with our own predicting layers. FC-T1 and FC_T2 as shown below.
Initial lower layers of the network learn very generic features from the pre-trained model. To achieve this initial layers weights of pre-trained models frozen and not updated during the training
Higher layers are used for learning task-specific features. Higher layers of pre-trained models are trainable or fine-tuned
Improves performance with less training time
We already have the data set created with image augmentation and we have created the base ResNet50 model.
we will now fine-tune Transfer learning with image augmentation using ResNet50. We do this by unfreezing a few of the last convolution blocks while keeping the first early conv blocks frozen. This will help us to learn very generic features using early layers. Higher layers of pre-trained models will be trainable or fine-tuned.
restnet.trainable = Trueset_trainable = Falsefor layer in restnet.layers:    if layer.name in ['res5c_branch2b', 'res5c_branch2c', 'activation_97']:        set_trainable = True    if set_trainable:        layer.trainable = True    else:        layer.trainable = Falselayers = [(layer, layer.name, layer.trainable) for layer in restnet.layers]pd.DataFrame(layers, columns=['Layer Type', 'Layer Name', 'Layer Trainable'])
We see that we have set the training for early layers of ResNet50 to false and the last few layers of ResNet50 are now trainable.
we now add our own fully connected layer and classifier on top of the ResNet50. We have already removed the last fully connected layer and the classifier layer from ResNet50
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, InputLayerfrom keras.models import Sequentialfrom keras import optimizersmodel_finetuned = Sequential()model_finetuned.add(restnet)model_finetuned.add(Dense(512, activation='relu', input_dim=input_shape))model_finetuned.add(Dropout(0.3))model_finetuned.add(Dense(512, activation='relu'))model_finetuned.add(Dropout(0.3))model_finetuned.add(Dense(1, activation='sigmoid'))model_finetuned.compile(loss='binary_crossentropy',              optimizer=optimizers.RMSprop(lr=1e-5),              metrics=['accuracy'])model_finetuned.summary()
We finally run the model
history_1 = model_finetuned.fit_generator(train_generator,                                   steps_per_epoch=100,                                   epochs=2,                                  validation_data=val_generator,                                   validation_steps=100,                                   verbose=1)
saving the weights of the fine-tuned model
model.save(‘cats_dogs_tlearn_finetune_img_aug_restnet50.h5’)
A Survey on Transfer Learning by Sinno Jialin Pan and Qiang Yang
A comprehensive hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning
GitHub for Hands-on Transfer Learning with Python