Deep Emotion Recognition based on Facial Expressions | by Stefanos Kanellopoulos | Towards Data Science

Motivated by the dissertation for my MSc in Robotics I decided to write a trilogy of articles. The aspiration behind this endeavor is to share the findings and the knowledge I have acquired through this magical journey. These three articles will mostly take the form of step-by-step coding tutorials while explaining on a theoretical level many technical choices that have been made.
Before we start implementing let’s talk about the field of Facial Expression Recognition (FER) a little bit:
FER is the scientific research area that deals with techniques and methods that try to identify/infer emotional states from facial expressions. In human communication, facial expressions play a crucial role in inferring emotions that could potentially help in understanding the intentions of others. According to different surveys [1, 2], verbal components convey only one-third of interpersonal communication, and non-verbal components convey two-thirds. The majority of messages related to attitudes and feelings lies in facial expressions. Hence, facial expressions have proven to play a vital role in the entire information exchange process. Expressions and emotions are indissolubly connected. Ekman and Friesen in [3] triggered the first wave of Basic Emotion Theory inspired studies on emotional expression. They used still photographs of prototypical emotional facial expressions and documented some degree of universality in the recognition and production of a limited set of “basic” emotions (happiness 😀, surprise 😮, fear 😨, disgust 🤮, sadness 😭, and anger 😡).
These 6 categories are going to be used for our task as well, and CK+48 [4] is the chosen dataset that will help us train and evaluate our model. The whole implementation took place on Google Colab using GPU acceleration. So, without further ado let’s dive into our task...
A) First things first, let’s import the necessary libraries as shown below.
Right after you store your dataset on your Google Drive, you have to mount your drive and go to the directory, which contains your dataset.
The next step is to load our data into the memory, this is why I created two auxiliary functions to make things easier. Below you can see their docstrings that describe their functionality (in order to avoid huge chunks of code in the article). Of course, you can find the full code on my GitHub account.
And this is how our data distribution looks like.
Note: Ideally, we should have equally distributed classes for our training since the model more frequently encounters examples of “surprise” compared to examples of “fear”. In the end, this will impair the model’s performance, as we will see later on that the majority of the misclassifications will be connected to the classes of “sadness” and “fear”.
B) The dataset I have used comprises 927 images of shape 48x48x3. Afterward, we randomly split the dataset into training (70%), validation (15%), and test (15%) sets (generally speaking, this is considered as a good rule of thumb to split our data).
An indicative print of the split is shown below.
X_train has shape: (648, 48, 48, 3) y_train has shape: (648, 6)  X_valid has shape: (139, 48, 48, 3) y_valid has shape: (139, 6)  X_test has shape: (140, 48, 48, 3) y_test has shape: (140, 6)X_train + X_valid + X_test = 927 samples in total
Note: It is very important for our validation and test set to be drawn from the same distribution. Dr. Andrew Ng explains the reason in a remarkable (and easy to digest) way in this video. The thing is that our data is coming from the same distribution (since it is created under lab constraints) but the largely unequal number of samples among some classes, combined with the relatively small amount of data can harm the efficiency of the model. Ideally, we should somehow “control” the ~140 samples for X_valid and X_test by dedicating ~23 different examples from each category.
C) Furthermore, data augmentation is going to be used only on the training samples in order to let the model encounter a variety of different examples since we don’t have so many examples at our disposal, and thus this will help it enhance its predictive performance.
Note: Not all parameters of ImageDataGenerator class are relevant. We have to be careful about that. For example, a vertical flip has no meaning for our task and presumably could harm the precision of the model.
Some examples of the augmentation applied on the training samples are shown below.
I created the function build_model that loads the EfficientNetB0 model pre-trained on ImageNet (without its original classifier) and adds 3 more layers at its top as you can see below. The idea of picking just 3 layers (and not something more complicated) came from the official documentation of Keras that shows how to fine-tune EfficientNet on an Image classification. Additionally, this function compiles the model with an Adam optimizer, a categorical cross-entropy loss function, and accuracy as a metric.
In my GitHub repo, I have uploaded the summary of the model as well as the graph of the model, so that you can take a look.
As for the training phase I used 3 very helpful callbacks:
ModelCheckPoint: Saves the weights of the model with which achieved the best validation accuracy.
EarlyStopping: If the validation accuracy does not get improved for 15 consecutive epochs, the training will be interrupted (this callback prevents overfitting)
ReduceLROnPlateaue: This is a scheduler that reduces the learning rate (by half) each time the validation accuracy is on a plateau.
The combination of the aforementioned callbacks assures that our model won’t overfit. Of course, we can also observe that through our learning curves that we will check in a while.
Note: As for the batch size, I experimented with a range of values. For batch sizes 16 and 32, the training time increased (and this is something that it does matter especially when the dataset is considerably large) while the validation accuracy wasn’t even higher compared to a batch of 64 samples. On the other hand, a batch of 128 samples, of course, accelerated the training procedure, however, not only the validation wasn’t better but also there were observed significant oscillations to the learning curves (something that could possibly hamper the convergence of the model).
Note: Two very important things that I realized during building this model are: (1) Despite the fact that in most cases pixel normalization is applied to the input images, EfficientNet does this anyway with its Rescale layers, and thus, further pixel-wise division with the value 255. would harm the performance of the model. (2) Despite the fact that the model takes as input tensors of shape (224, 224, 3), I realized that better performance achieved when I modified the Input Layer in order to receive tensors of shape (48, 48, 3), instead of rescaling the dataset to 224x224. The only relevant source I managed to find was from Adrian Rosebrock in one of his articles on pyimagesearch.com.
In the end, the model needed only 46 epochs to converge and the training procedure was interrupted from our callback. The corresponding learning curves are shown below.
The model seems to fit quite well the new data with a fairly small generalization gap. Both training and validation accuracy approximated 98-99% accuracy. However, the weights of the best version of the model will automatically be loaded.
Finally, we are going to test our model on samples that it has not seen again. This set is the test set that is drawn from our initial dataset. The model achieved 96.43% accuracy, and the predictions are depicted through a confusion matrix so that we can thoroughly check the misclassifications of the model.
As you can observe from this matrix, the class “sadness” essentially causes the problem. It is absolutely reasonable though, due to the small number of examples that are available. Furthermore do not forget that even for a human, facial expressions that express sadness, fear, disgust, or anger might be confusing anyway, and this can become obvious if you check the augmented samples of facial expressions above, and try to predict the class of each expression on your own. You will realize that there is inherent difficulty on this task even for humans.
You can also see some random predictions of the model with a barplot on the right side that shows the confidence of each prediction.
In this article, we saw how we can build a classifier of facial expressions that could predict emotions. For this dataset (CK+48), achieving 96.43% accuracy is near to the state-of-the-art performance, and in the next article of this trilogy, we are going to apply a visualization technique (GradCam) that reveals the regions of an image that play a crucial role for a model in order to infer, by applying heatmaps onto the original images. Such techniques are very important for deep learning approaches because they assist in the model’s explainability and interpretability. So, at our next meeting, we will see how we can interpret our model’s inaccuracies and how we can possibly improve it.
Thank you so much for your time! (The full project can be found here).
[1] Albert Mehrabian. Communication without words. Communication theory, pages 193–200, 2008.
[2] Kathrin Kaulard, DouglasWCunningham, Heinrich H Bülthoff, and ChristianWallraven. The mpi facial expression database — a validated database of emotional and conversational facial expressions. PloS one, 7(3), 2012.
[3] Paul Ekman, E Richard Sorenson, and Wallace V Friesen. Pan-cultural elements in facial displays of emotion. Science, 164(3875):86–88, 1969.
[4] Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 ieee computer society conference on computer vision and pattern recognition-workshops, pages 94–101. IEEE, 2010.