Detecting Human Facial Emotions - Multistage transfer learning approach
By: Sudarshan Vaidya
Tags: Data Science, Facial Expressions, Pytorch, Machine Learning, Emotion Recognition

Detecting Human Emotions - Facial Expression Recognition

That's me - attempting a pretension of contempt

Introduction

One of the growing applications of Machine Learning in Computer Vision field has been human sentiment recognition on the basis of visual (face), audio (speech) and text.

Facial expression is one of the fundamental means to communicate effectively with people. The ability to understand the facial emotion of others is a key to successful communication.

There are innumerable ways in which sentiment recognition can be applied gainfully in diverse businesses. Some of which could be:

- Psycho analysis (medical)

- Non verbal customer feedback (Retail / Marketing)

- Candidate selection (Human resources)

- Security (agitation threat detection)

- Invigilation (Education)

- Aid for differently-abled

Limitations and Alternatives

The neural network based Machine Learning approach for facial emotion recognition usually has a limitation that there is a focus only on learning one emotion at a time, and that emotion needs to be expressive enough to distinguish from other emotions — while in real life human expressions could be mixed and subtle. There is a special system for coding different facial expression parts into some “action units” — Facial action coding system (FACS). https://en.wikipedia.org/wiki/Facial_Action_Coding_System

Source: Wikipedia

However, with the advent of large number of facial expression datasets for training, the accuracy of machine learning algorithms have been satisfactory enough for general applications.

In this article, I have tried to describe how I designed, adapted and implemented my first human face sentiment detection.

Challenges in facial expression classification

Recognition of facial emotions by computer with high recognition rate is still a challenging task.

Before we go into details, it is good to know the major obstacles in facial emotion Recognition:

1. Emotions are subjective, people would interpret it differently. It is hard to define the notion of emotions. E.g. although overall tents may remain the same, different people ‘appear’ to display the same emotion in widely different ways.

2. It is not possible to measure intensity of emotions quantitatively.

3. The expressiveness of people could be very different. It becomes difficult to gauge emotions of a less expressive person.

4. There may not be a linear relationship between emotions and facial expressions. E.g. it definitely varies from person to person, as also for a given person, it may vary over time.

5. There could be multiple emotions at a single instance, and the resultant facial expression may not be the real representative of the pre-dominant expression.

6. Annotating images is challenging. Perceived emotion from a still image may have a historical context, which is unknown to the observer.

7. Annotating images at large scale requires high human and time cost. The annotation result has to be evaluated by multiple individuals due to its subjectivity.

8. Since the emotions are overlapping in some aspects, even the human accuracy is low, e.g. sometimes there is a thin line between fear and surprise, boredom and sadness etc.

These challenges can be addressed by multimodal emotion detection, i.e. using speech, image / video, text and sensory data. But that is beyond the scope of this discussion.

Solution Approach

Before we begin, let us briefly look at the salient features, approach followed and tools used:

1. Standard datasets: Use of multiple face emotion datasets — 33,000 standard images

2. Generated datasets: 5,500 real world images from Google, Unsplash, Flickr

3. Training approach: Three step approach

4. Architecture: Resnet50 using PyTorch

5. Transfer learning: Pretrained on ImageNet

6. Face detection: MTCNN

7. Emotion categorization: 8 emotions — anger, contempt, disgust, fear, happiness, neutrality, sadness and surprise

8. Accuracy improvement with corrective re-annotation of images

9. End use: Continuous facial emotion recognition on live webcam video

Data gathering and Pre-processing

1. Standard datasets like FER, CKPlus and KDEF were used, which provided about 33,000 images.

2. To augment the images available for training from the real world, images containing human face from several other sources like Google image search, Unsplash and Flickr were gathered.

3. In cases where APIs were not provided, or did not work, tools like “Fatkun batch downloder” were employed to download images in thousands.

4. Real world images were processed to detect human images and crop them for faces

5. Cropped face images were then converted into standard grayscale 224 X 224 PNG images

Summary table of data sources considered:

Data sources considered

Notes:

1. Contempt and Neutral categories were not present in some data sources, but other available categorizations were utilized.

2. Indian Movies Face database was excluded as the resolution of images was not satisfactory, and required prohibitive efforts for manual inspection and annotation.

3. Self created images using own and friends’ pictures was abandoned as the volume would have been too less compared to the efforts required.

Why PyTorch ?

I tried using PyTorch for the first time. I have worked with TensorFlow / Keras several times, and wanted to discover PyTorch — hearing about its ease of learning and usability for building rapid prototypes.

Multistage transfer learning approach

The base model used for this project is ResNet50 that is pre-trained on ImageNet.

Three levels of training has been used.

1. Grayscale training: Adapt an ImageNet-pretrained Resnet50 model to accept grayscale images: Model is designed to work on Grayscale images for simplicity. Instead of using default weight initializations in ResNet50 input layer, these weights have been obtained by unsupervised training on a small subset of ImageNet dataset (~ 35k images) to yield same output as colored input images.

2. Stage 1 training: Train the model on images consolidated from three sources — FER, CK+ and KDEF: The face images from FER, CK+ and KDEF datasets are standard, or created under laboratory conditions. Total number of images is more than 33K. Hence they are used to retrain the entire ResNet50 model (all layers) with a low learning rate.

3. Stage 2 training: Train partially with real word images: The real life face images obtained from the internet are manually annotated and used for partial retraining of the ResNet50 model. Partial retraining was tried for top 3, top 4 and top 5 layers.

Overcoming the Mis-categorization Challenge

On manual inspection, it was found that several images in the standard datasets were mis-categorized. Some examples are given below.

So I decided to run an experiment to check if corrective re-annotation leads to any improvement in the accuracy.

1. For Stage 1 training, corrective re-annotation meant that the emotion mis-classification in the existing datasets was manually corrected. It took efforts of 12 hours for ~ 35k images.

2. For Stage 2 training, corrective re-annotation meant that the emotion classification which was done by observer 1 was refined by another observer 2. Also, the output model from Stage 1 training after corrective re-annotation was used as the base model used for Stage 2 training.

Result: It turned out that corrective re-annotation led to an increase in the Stage 1 training validation accuracy from 70% to 75%, and Stage 2 training validation accuracy from 59% to 70%.

This is how the validation accuracies for the two training stages improved. Overall, there was an increase of more than 11%.

Validation accuracy before and after corrective re-annotation

Data references

1. Mini ImageNet Dataset: : https://www.kaggle.com/ifigotin/imagenetmini-1000

2. FER Dataset: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge

3. CK Plus Dataset: https://github.com/WuJie1010/Facial-Expression-Recognition.Pytorch/tree/master/CK%2B48

4. KDEF Dataset: https://www.kdef.se/download-2/register.html

5. Corrective re-annotation: https://www.kaggle.com/sudarshanvaidya/corrective-reannotation-of-fer-ck-kdef

6. Natural Human Face Images for Emotion Recognition: https://www.kaggle.com/sudarshanvaidya/random-images-for-face-emotion-recognition

Demo Videos

Finally, this is how it turned out to be after facial expression recognition was applied on images of live video capture.

I tried pretending to be angry, disgusted, surprised, sad, neutral, contempt and scared. My daughter also helped to bring in some variety - though she always looked cute and happy most of the time.

1. Demo video 1: Self https://youtu.be/qXO95-E4ZPk

2. Demo video 2: Self & daughter https://youtu.be/EZTgSxnUppM