Facial Landmark Detection for Occluded Angled Faces | Towards Data Science

Facial landmarks detection or facial keypoints detection has a lot of uses in computer vision like face alignment, drowsiness detection, Snapchat filters to name a few. The most widely known model for this task is Dlib’s 68 keypoints landmark predictor which gives very good results in real-time. But the problem starts when the face is occluded or at an angle to the camera. Getting accurate results at angles is crucial for tasks like head pose estimation. So in this article, I will introduce a lesser-known Tensorflow model that can achieve this mission.
Now, this is finding drawbacks just because I want to but in essence, installing Dlib in Windows is a little tough and one needs to install Cmake and some other applications to do so. If you are using Anaconda and try using conda install, then the version for Windows is outdated as well and does not support Python>3.5. If you are using any other operating system then you will be fine.
Instead of writing what’s wrong, I’ll just show it to you.
All right enough of maligning Dlib let’s move on to the real stuff.
We will require Tensorflow 2 and OpenCV for this task.
# pip installpip install tensorflowpip install opencv# conda installconda install -c conda-forge tensorflowconda install -c conda-forge opencv
Our first step is to find the faces in the images on which we can find facial landmarks. For this task, we will be using a Caffe model of OpenCV’s DNN module. If you are wondering how it fares against other models like Haar Cascades or Dlib’s frontal face detector or you want to know more about it in-depth then you can refer to this article:
towardsdatascience.com
You can download the required models from my GitHub repository.
import cv2import numpy as npmodelFile = "models/res10_300x300_ssd_iter_140000.caffemodel"configFile = "models/deploy.prototxt.txt"net = cv2.dnn.readNetFromCaffe(configFile, modelFile)img = cv2.imread('test.jpg')h, w = img.shape[:2]blob = cv2.dnn.blobFromImage(cv2.resize(img, (300, 300)), 1.0,(300, 300), (104.0, 117.0, 123.0))net.setInput(blob)faces = net.forward()#to draw faces on imagefor i in range(faces.shape[2]):        confidence = faces[0, 0, i, 2]        if confidence > 0.5:            box = faces[0, 0, i, 3:7] * np.array([w, h, w, h])            (x, y, x1, y1) = box.astype("int")            cv2.rectangle(img, (x, y), (x1, y1), (0, 0, 255), 2)
Load the network using cv2.dnn.readNetFromCaffe and pass the model's layers and weights as its arguments. It performs best on images resized to 300x300.
We will be using a facial landmark detector provided by Yin Guobing in this Github repo. It also gives 68 landmarks and it is a Tensorflow CNN trained on 5 datasets! The pre-trained model can be found here. The author has also written a series of posts explaining the background, dataset, preprocessing, model architecture, training, and deployment that can be found here. I have provided their outlines here, but I would strongly encourage you to read them.
In the first of those series, he describes the problem of stability of facial landmarks in videos followed by labeling out the existing solutions like OpenFace and Dlib’s facial landmark detection along with the datasets available. The third article is all about data preprocessing and making it ready to use. In the next two articles, the work is to extract the faces and apply facial landmarks on it to make it ready to train a CNN and store them as TFRecord files. In the sixth article, a model is trained using Tensorflow. In the final article, the model is exported as an API and shown how to use it in Python.
We need the coordinates of the faces that acts as the region of interest and is extracted from the image. Then it is converted to a square shape of size 128x128 and passed to the model which returns the 68 key points which can then be normalized to the dimensions of the original image.
The complete code used for this piece of work:
Now, that you know it can perform well on side-faces it can be utilized to make a head-pose estimator. If you want to do you can refer to this article:
towardsdatascience.com
One of the major selling points of Dlib was its speed. Let’s see how they compare on my i5 processor (yeah 😔).
Dlib gives ~11.5 FPS and the landmark prediction step takes around 0.005 seconds.
The Tensorflow model gives ~7.2 FPS and the landmark prediction step takes around 0.05 seconds.
So if speed is the main concern and occluded or angled faces not so much then Dlib might be better suited for you otherwise I feel that the Tensorflow model reigns supreme without compromising a lot on speed.
You find the properly documented code for both face detection and landmark detection here on GitHub, where I have tried to make an AI for online proctoring.