Emotion Recognition
By: Ketan Vaidya
Tags: Convolutional Network, Opencv, Realtime, Emotion Recognition, Deep Learning

Emotion Recognition

It’s fascinating how much progress deep learning has made in just a few years that we can use Convolutional Neural Networks for such a large variety of tasks.

Today I’m going to show you one such exciting task to perform with CNNs which is Emotion Recognition. You have to just say hi to your camera and it will predict the emotion on your face. Just like that!

Photo by Tengyart on Unsplash

So let’s get started with importing the libraries required to perform this task.

1. Dataset

For this project, I used the FER 2013 dataset available on Kaggle. This was the only easily available dataset I could find for this task. The dataset contains CSV files that map the emotion labels to the respective pixel values of the image at hand.

It has 7 emotions/classes (0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral) and as visible in the overview below, the classes are a little skewed.

The dataset looks something like :

An overview of how the dataset looks

Given the look of the dataset, we will need to preprocess these pixel values to convert them into meaningful images.

2. Preprocessing the pixel values

We will need to load the pixel values into Google Colab and convert them into their image form. For this purpose, I have written the function preprocess_pixels to tackle this task.

The function takes to input the pixel_data( pixels column of the dataset) and converts each of the entry to from a string to an array of integers which is reshaped into the images which we will train on.

As I had mentioned earlier that, or dataset is a little bit skewed i.e. some classes have a lot more examples than some others.

One way to overcome this is by using RandomOverSampler which oversamples the classes which are in lesser numbers in a dataset. It basically randomly duplicates examples in the minority classes, which helps in reducing the imbalance in the dataset.

The sampling_strategy ‘auto’ means that all the classes will end up with the same number of instances.

Now let's convert this into the image form, using the function defined earlier, and split the data into train and test.

Let’s see how an image looks from our dataset!

Now the time to define our model which will train our images.

3. Defining the Model

I have defined a CNN model here, but you can use the many available pre-trained models like VGG-16, Inception V3, or ResNet for this task. You can use transfer learning which will be useful to use here.

4. Training the model

Before training, the image emotion labels need to be one-hot encoded, we do that using to_categorical.

Now we will fit the training set to the model using 30 epochs

At the end of 30 epochs, the training accuracy is 95.6% while the validation accuracy is 82.81% which is a good value obtained as the model used wasn't the most complex and we had also used oversampling to account or skewness of the data.

5. Validating the Model

After we have trained the model, the model weights were saved and model.predict was used on the validation set. It returns the ‘preds’ vectors for each image in the dataset.

A function get_class is defined which converts the preds into a vector of only image labels. It uses np.argmax which returns the index of the maximum value in the vector preds[i].

A label dictionary mapping the integer labels to their emotions is defined.

And now an image out of the dataset is taken to check the model’s accuracy.

The following was the output after running the cell above

We can tell that the image above depicts ‘happiness’, and the model correctly predicted the emotion to be ‘happiness’ which is a good thing!

Now we will implement this model in real-time to predict emotions using the webcam.

6. Implementing the model in Real-Time

This is a simple extension or task which can be easily implemented using OpenCV which is a library focused on real-time applications for Computer Vision. OpenCV stands for Open Source Computer Vision Library.

For this task, we will use haar cascade classifiers which is an effective object-detection method, and here will be used to detect the face in the real-time setting.

First, we will download the file ‘haarcascade_frontalface_default.xml’, which is a file containing the features required to detect the frontal face. This file can be downloaded from Kaggle and also available on GitHub.

cv2.VideoCapture(0) starts the webcam and captures images from the webcam which are passed into the cascade classifier to detect the face in the image.

The detected face is then converted to a grayscale image and resized into a shape that is compatible with our model. Then it is passed to the model for predicting the emotion.

The emotion label is displayed on the screen along with a square box encapsulating the face.

Below are the images I obtained by testing the model in real-time using a webcam.

Image by Author

The black font might not be visible at first, but if you look closely the emotions are displayed :D.

The model showed pretty good results for real-time implementation too except for emotions like fear and disgust which had the least amount of training examples.

So in this article, we implemented an emotion recognition model using CNN and the FER 2013 dataset. After training the model, the model was validated, and also a real-time implementation of the model was done successfully using OpenCV and HaarCascade classifiers.

Hope you enjoy reading this article!

Thank You!