Real-time Age, Gender and Emotion Prediction from Webcam with Keras and OpenCV | by Sun Weiran | Towards Data Science

In the era of Covid-19, we become more reliant on virtual interactions such as Zoom meetings / Teams chat. These livestream webcam videos have become a rich data source to explore. This article will explore the use case of age, gender and emotion prediction which could facilitate sales person to understand their customers better, for example.
As shown in above figure, the implementation contains 4 main steps:
Receive input frame from the webcamIdentify faces in the webcam and prepare these images for the 3 deep learning models, i.e. age, gender and emotion modelsSend processed faces to the models and receive prediction outcomesRender prediction outcomes with bounding boxes to screen
Receive input frame from the webcam
Identify faces in the webcam and prepare these images for the 3 deep learning models, i.e. age, gender and emotion models
Send processed faces to the models and receive prediction outcomes
Render prediction outcomes with bounding boxes to screen
In this implementation, we will be using one of the state-of-the-art facial recognition models, MTCNN for step 2. It has a stable Python release based on Keras available at here.
For step 3, we will be training our own customized models. However, for less efforts and better accuracy, you might want to consider transfer learning techniques. Many pre-trained models including, VGG-face, FaceNet, GoogLeNet are available. Note that these pre-trained models might have different input size requirements. Thus faces identified from step 2 need to be processed accordingly.
Facial recognition has become a mature application of deep learning in recent years. Many algorithms have been proposed for fast and accurate face detection in images / videos. MTCNN is one of these that is based on FaceNet.
In Python’s implementation, the model has been pre-trained and optimized therefore we can make use of the model directly. Nevertheless it’s still important to understand the model’s output.
[    {        'box': [277, 90, 48, 63],        'keypoints':        {            'nose': (303, 131),            'mouth_right': (313, 141),            'right_eye': (314, 114),            'left_eye': (291, 117),            'mouth_left': (296, 143)        },        'confidence': 0.99851983785629272    }]
for every image, MTCNN returns a list of dictionaries which every dictionary represents a face detected in the photo. Every face is represented as a bounding box — a rectangle that surrounds the face.
box: [x, y, width, height], x and y are coordinates of the top left corner of the bounding box
keypoints: dictionary of facial landmark points detected
confidence: model’s confidence score on the face detected with 1 being the most confident.
The emotion model is trained from CKPlus Facial Emotion dataset. This dataset contains 981 images from 7 emotion classes: anger, contempt, disgust, fear, happy, sadness and surprise. Each image is in grey scale and has fixed size of 48*48
The age and gender models are trained from UTKface dataset. The dataset contains more than 20k images. Each image is labelled with age, gender and race. Both full photos and cropped face photos are available for download. For this article, we will be using the full photos and implement our own face alignment method for better accuracy.
We need to crop face from the full photo using MTCNN or any other facial recognition models. However, most of these algorithms give bounding boxes of varying shapes based on the size and position of the face detected.
Deep learning models require input images to have standardized sizes (caveat: not applicable for fully convolutional network, out-of-scope for this article). Hence resizing of the cropped faces becomes necessary. Direct resizing is the most common and straightforward method with obvious downside — face deformation. As illustrated with the below figure, face becomes significantly wider after direct resizing. This will negatively affect our model performances.
An ideal cropped face photo should have the face located at the center with no distortion and the required size. If the required size is a square, the following method does the trick.
Obtain facial bounding box from MTCNNFind the center point of the bounding boxFind the maximum between the height and width of the bounding boxDraw new bounding box based on the center and maximum side lengthResize the cropped face from the new bounding box to the required size
Obtain facial bounding box from MTCNN
Find the center point of the bounding box
Find the maximum between the height and width of the bounding box
Draw new bounding box based on the center and maximum side length
Resize the cropped face from the new bounding box to the required size
If the required size is not a square, step 3 and 4 need to be adjusted to maintain the same ratio as the required sides.
For age and gender model, we will use centered resizing methods on the full photos with MTCNN. Required input size is set to be (224, 224, 3) for both models.
It’s not the most ideal dataset for emotion prediction due to its image format (grey scale) and small volume. The advantage is that all images are nicely cropped and aligned thus good for quick prototyping.
One note for this dataset: for each emotion class, individual faces are repeated 3 times. Hence if train/test split is done randomly, target leakage will happen. Recommend to split based on subjects or to remove duplicates before randomly splitting.
Among the three targets, age is the most difficult task. Sometimes even people make mistakes guessing other’s age. Therefore we will need a deeper model for age prediction. In general, these are typical convolutional neural networks. Please note all of these model structures are neither tuned nor optimized. Purpose of the article does not include fine-tuning deep learning models.
Basically, openCV captures video from your webcam (line 2). For every frame, it will convert it to RGB format (line 19). This RGB frame will be sent to detect_face function (line 22), which firstly detects all the faces in the frame using MTCNN and for every face, predicts using the 3 trained models to generate outcomes. These outcomes are returned together with the face bounding box locations (top, right, bottom, left).
OpenCV then makes use of the bounding box locations to draw rectangle on the frame (line 27) and display prediction outcomes in text(line 29 — line 32).
Implementation of the detect_face function can be found in the source code. Note that since emotion model is trained from grey-scale images, RGB image needs to be grey-scaled before being predicted by emotion model.
Better dataset for emotion prediction
Due to computational resource constraint, only 5k images from UTKface dataset were used for age/gender model training. Model performance could be improved by using more images
Image augmentation
Transfer learning / tuning model architecture
Thanks for reading!
Happy New Year!