VGG-16 Transfer Learning in Classifying Log-Mel Spectrogram Images | by Muriel Kosaka | Towards Data Science

As a follow-up to my previous post, I will be applying transfer learning to the RAVDESS Audio Dataset in hopes to improve the model’s accuracy. To review, transfer learning is a deep learning approach in which a model that has been trained on one task is used as a starting point to train a model for a similar task. In this post by DJ Sarkar, he provides a great guide in understanding transfer learning with examples.
We will first try to use the VGG-16 pretrained model as a feature extractor on our dataset, which is where we freeze the convolution blocks of a pretrained model and modify the dense layers. Then we will try the pretrained model with Image Augmentation.
So let’s get started!
After importing the necessary libraries, we must import the training and test images with labels.
# For training set onlyimport globangry = glob.glob('/content/drive/My_Drive/train_logmel/angry/*.*')calm = glob.glob('/content/drive/My_Drive/train_logmel/calm/*.*')disgust = glob.glob('/content/drive/My_Drive/train_logmel/disgust/*.*')fearful = glob.glob('/content/drive/My_Drive/train_logmel/fearful/*.*')happy = glob.glob('/content/drive/My_Drive/train_logmel/happy/*.*')neutral = glob.glob('/content/drive/My_Drive/train_logmel/neutral/*.*')sad = glob.glob('/content/drive/My_Drive/train_logmel/sad/*.*')surprised = glob.glob('/content/drive/My_Drive/train_logmel/surprised/*.*')data = []labels = []for i in angry:       image=tf.keras.preprocessing.image.load_img(i, color_mode='rgb',     target_size= (224,224))    image=np.array(image)    data.append(image)    labels.append('Angry')for i in calm:       image=tf.keras.preprocessing.image.load_img(i, color_mode='rgb',     target_size= (224,224))    image=np.array(image)    data.append(image)    labels.append('Calm')for i in disgust:       image=tf.keras.preprocessing.image.load_img(i, color_mode='rgb',     target_size= (224,224))    image=np.array(image)    data.append(image)    labels.append('Disgust')for i in fearful:       image=tf.keras.preprocessing.image.load_img(i, color_mode='rgb',     target_size= (224,224))    image=np.array(image)    data.append(image)    labels.append('Fearful')for i in happy:       image=tf.keras.preprocessing.image.load_img(i, color_mode='rgb',     target_size= (224,224))    image=np.array(image)    data.append(image)    labels.append('Happy')for i in neutral:       image=tf.keras.preprocessing.image.load_img(i, color_mode='rgb',     target_size= (224,224))    image=np.array(image)    data.append(image)    labels.append('Neutral')for i in sad:       image=tf.keras.preprocessing.image.load_img(i, color_mode='rgb',     target_size= (224,224))    image=np.array(image)    data.append(image)    labels.append('Sad')for i in surprised:       image=tf.keras.preprocessing.image.load_img(i, color_mode='rgb',     target_size= (224,224))    image=np.array(image)    data.append(image)    labels.append('Surprised')train_data = np.array(data)train_labels = np.array(labels)
Let’s see a sample of images from the training set:
Normalize data
Normalize data
X_train = X_train.astype('float32')X_test = X_test.astype('float32')X_train /= 255X_test /= 255
2. One-hot encode target classes
lb = LabelEncoder()y_train = np_utils.to_categorical(lb.fit_transform(y_train))y_test = np_utils.to_categorical(lb.fit_transform(y_test))
First we import VGG16 and set the necessary arguments:
from keras.applications import VGG16vgg_model = VGG16(weights='imagenet',include_top=False, input_shape=(224, 224, 3))
weights = ‘imagenet’ : use the pretrained weights instead of training the model from scratch
include_top=False : we want to load the model without the classifier layers and add our own
input_shape=(224,224,3) : specify the preferred shape of the images in our dataset
Next we freeze the convolution blocks:
for layer in vgg_model.layers:layer.trainable = False# Make sure you have frozen the correct layersfor i, layer in enumerate(vgg_model.layers):    print(i, layer.name, layer.trainable)
Then create our Dense layers:
x = vgg_model.outputx = Flatten()(x) # Flatten dimensions to for use in FC layersx = Dense(512, activation='relu')(x)x = Dropout(0.5)(x) # Dropout layer to reduce overfittingx = Dense(256, activation='relu')(x)x = Dense(8, activation='softmax')(x) # Softmax for multiclasstransfer_model = Model(inputs=vgg_model.input, outputs=x)
And lastly, we compile and fit the model:
learning_rate= 5e-5transfer_model.compile(loss="categorical_crossentropy", optimizer=optimizers.Adam(lr=learning_rate), metrics=["accuracy"])history = transfer_model.fit(X_train, y_train, batch_size = 1, epochs=50, validation_data=(X_test,y_test))
After 50 epochs, we achieved an accuracy score of 69%.
This performed slightly better than previous models, but there has to be a way to improve model accuracy! Let’s try using the VGG-16 as a feature extractor with Image Augmentation. Image Augmentation is a great way to add more training data when dealing with a small dataset.
Using ImageDataGenerator we can augment the images:
train_datagen = ImageDataGenerator(rescale=1./255, zoom_range=0.3, rotation_range=40, width_shift_range=0.3, height_shift_range=0.3, shear_range=0.3, horizontal_flip=True, fill_mode="nearest")train_generator = train_datagen.flow(train_data, train_lb, batch_size=1)val_datagen = ImageDataGenerator(rescale=1./255)val_generator = val_datagen.flow(test_data,val_lb,batch_size=1)
Next, we will build our deep learning model, compile the model, then fit it:
x = vgg_model.outputx = Flatten()(x) # Flatten dimensions to for use in FC layersx = Dense(512, activation='relu')(x)x = Dropout(0.5)(x) # Dropout layer to reduce overfittingx = Dense(256, activation='relu')(x)x = Dense(8, activation='softmax')(x) # Softmax for multiclasstransfer_model = Model(inputs=vgg_model.input, outputs=x)learning_rate= 5e-5transfer_model.compile(loss="sparse_categorical_crossentropy", optimizer=keras.optimizers.Adam(lr=learning_rate), metrics=["accuracy"])history = transfer_model.fit_generator(train_generator, validation_data=val_generator, epochs=100, shuffle=True, verbose=1)
After 100 epochs, we achieved an accuracy score of 78%
As you can see this did not perform much better than our previous model and the current model is overfitting the training data.
I will look into other pretrained models (such as Inception_V3 and Resnet50) and explore fine-tuning rather than using pretrained models as feature extractors. Thank you for reading! :)