Fine-Tuning Pre-trained Model VGG-16 | by Muriel Kosaka | Towards Data Science

In my previous article, I explored using the pre-trained model VGG-16 as a feature extractor for transfer learning on the RAVDESS Audio Dataset. As a newcomer to Data Science, I read through articles here on Medium and came across this handy article by Pedro Marcelino in which he describes the process of transfer learning, but most insightful was the three strategies in which one can fine-tune their selected pre-trained model. Pedro Marcelino also provides a helpful Size Similarity Matrix to help determine which strategy to use. After reading his article, I’ve come to realize that rather than using pre-trained models as feature extractors, I should be fine-tuning the model by training some layers and leaving other layers frozen as my dataset is small (1,440 files) and not similar to the VGG-16 model dataset. Here I will explore this type of fine-tuning of the VGG-16 pre-trained model on the RAVDESS Audio Dataset and determine its effect on model accuracy.
After importing the necessary libraries, our train/test set, and preprocessing the data (described here), we dive into modeling:
First, import VGG16 and pass the necessary arguments:
First, import VGG16 and pass the necessary arguments:
from keras.applications import VGG16vgg_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
2. Next, we set some layers frozen, I decided to unfreeze the last block so that their weights get updated in each epoch
# Freeze four convolution blocksfor layer in vgg_model.layers[:15]:    layer.trainable = False# Make sure you have frozen the correct layersfor i, layer in enumerate(vgg_model.layers):    print(i, layer.name, layer.trainable)
Perfect, so we will be training our dataset on the last four layers of the pre-trained VGG-16 model.
3) I used the same model architecture from my previous VGG-16 model as a feature extractor:
x = vgg_model.outputx = Flatten()(x) # Flatten dimensions to for use in FC layersx = Dense(512, activation='relu')(x)x = Dropout(0.5)(x) # Dropout layer to reduce overfittingx = Dense(256, activation='relu')(x)x = Dense(8, activation='softmax')(x) # Softmax for multiclasstransfer_model = Model(inputs=vgg_model.input, outputs=x)
4) Let’s set some callbacks, such as ReduceLROnPlateau and ModelCheckpoint. ReduceLROnPlateau is helpful especially in fine-tuning our model because as described by Pedro Marcelino, high learning rates increase the risk of losing previous knowledge so it is best to set a low learning rate, and with ReduceLROnPlateau, this can help us out with that! ModelCheckpoint is always helpful since it allows us to define where to checkpoint our model weights.
from keras.callbacks import ReduceLROnPlateaulr_reduce = ReduceLROnPlateau(monitor='val_accuracy', factor=0.6, patience=8, verbose=1, mode='max', min_lr=5e-5)checkpoint = ModelCheckpoint('vgg16_finetune.h15', monitor= 'val_accuracy', mode= 'max', save_best_only = True, verbose= 1)
4) Next, we compile and fit our model
from tensorflow.keras import layers, models, Model, optimizerslearning_rate= 5e-5transfer_model.compile(loss="categorical_crossentropy", optimizer=optimizers.Adam(lr=learning_rate), metrics=["accuracy"])history = transfer_model.fit(X_train, y_train, batch_size = 1, epochs=50, validation_data=(X_test,y_test), callbacks=[lr_reduce,checkpoint])
As we can see the model is largely overfitting to the training data. After 50 epochs, our model achieved an accuracy of 78% which is 9% higher than our previous classifier, where we used the pre-trained VGG-16 model used as a feature extractor, but performed the same as our pre-trained VGG-16 model used as a feature extractor with image augmentation.
Now let's try fine-tuning VGG-16 model with image augmentation to see if that will improve model accuracy. Using the same VGG-16 model object stored in the transfer_model variable from our previous model and unfreeze the fifth convolution block while keeping the first four blocks frozen.
for layer in vgg_model.layers[:15]:layer.trainable = Falsex = vgg_model.outputx = Flatten()(x) # Flatten dimensions to for use in FC layersx = Dense(512, activation='relu')(x)x = Dropout(0.5)(x) # Dropout layer to reduce overfittingx = Dense(256, activation='relu')(x)x = Dense(8, activation='softmax')(x) # Softmax for multiclasstransfer_model = Model(inputs=vgg_model.input, outputs=x)for i, layer in enumerate(transfer_model.layers):print(i, layer.name, layer.trainable)
Great, so the fifth convolution block is now trainable and we’ve added our own classifier. Now, we will augment our images using ImageDataGenerator of Keras’ image preprocessing module, fit it to our training set, compile the model, then fit the model.
#Augment imagestrain_datagen = ImageDataGenerator(zoom_range=0.2, rotation_range=30, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2)#Fit augmentation to training imagestrain_generator = train_datagen.flow(X_train,y_train,batch_size=1)#Compile modeltransfer_model.compile(loss="categorical_crossentropy", optimizer='adam', metrics=["accuracy"])#Fit modelhistory = transfer_model.fit_generator(train_generator, validation_data=(X_test,y_test), epochs=100, shuffle=True, callbacks=[lr_reduce],verbose=1)
After 100 epochs, we achieved an accuracy score of 70%, that is an 8% decrease from our previous model and performed the same as our VGG-16 model as a feature extractor (*insert huge sad face here*).
As can be clearly seen, the model is overfitting to the training data and accuracy seems to plateau after about 40 epochs. With fine-tuning, I did not see much improvement in model accuracy over using the pre-trained model as a feature extractor which I did not expect as per the dissimilarity and smaller dataset I am using compared to the VGG-16 model.
I have tried to adjust the parameters of the ImageDataGenerator class and was unsuccessful in improving model accuracy. If anyone has any suggestions as to how/if my VGG-19 model fine-tuned with image augmentation can improve, please let me know! Thank you for reading :)