Face Landmark Detection using Python | Towards Data Science

Face landmark detection is a computer vision task where we want to detect and track keypoints from a human face. This task applies to many problems.
For example, we can use the keypoints for detecting a human’s head pose position and rotation. With that, we can track whether a driver is paying attention or not. Also, we can use the keypoints for applying an augmented reality easier. And there are so many solutions that we can generate based on this task.
Thankfully, we don’t have to understand the concepts of face landmark detection in detail. We can use the prebuilt library like dlib, OpenCV, and mediapipe. In this article, I will show you how to implement face landmark detection with dlib and mediapipe.
Without further, let’s get started!
Dlib is a library for applying machine learning and computer vision solutions. This library is based on the C++ language, but we can use a language like Python for using the library. One of the solutions that we can apply by using this library is face landmark detection. Now let’s get into the implementation.
Installing a library can become a problem. If we don’t have a good guide, installing the library can take several days. Dlib is one of them. Because it uses C++ as the primary language, we have to install C++ tools for installing the library.
There are several steps that we should do to install it. Here are the steps:
First, install the CMake. You can download the software here. If you are using Windows, please locate the CMake file path first. Then, set the path to the executable path on the environment variable.
Then, install Visual Studio with the C++ dependencies to it. You can download the software here. For the dependencies, you can look at this screenshot below:
After you install the Visual Studio, the next step is to install the Python. To make your installation simpler, I recommend you for installing Anaconda. You can download it here. For the Python version, I recommend you for using the 3.6.6 version to avoid any errors.
Lastly, install the CMake, dlib, and OpenCV library by using pip. Here is the command for doing that:
pip install cmakepip install dlib pip install opencv-contrib-python
After we’ve installed the libraries, the next step is to import them into our code. We will import OpenCV for retrieving inputs from the webcam, NumPy for numerical computation, and Dlib for detecting keypoints from a face.
Here is the code for doing that:
Now let’s initialize several variables. There are three must need variables that we will initialize:
A detector for detecting one or more faces. We set the dlib.get_frontal_face_detector function inside the variable.
A predictor for detecting keypoints from faces. We set the dlib.shape_predictor function inside the variable. This function needs a pretrained model location as the parameter, which you can download here.
The cv2.VideoCapture object for capturing images from the webcam. Also, we set a parameter with value 0 for capturing images from a webcam.
Let’s write this code for initializing variables:
As you can see from above, we initialize the face landmark detector by using the pretrained model. The model is based on ensemble regression trees because the model will predict continuous numbers. You can read the details about the model here.
That model is trained on the iBUG-300 W dataset, where it contains images and their corresponding 68 face landmark points. In general, those landmark points belong to the nose, the eyes, the mouth, and the edge of a face. You can download the dataset here.
Here is the visualization of the face landmark locations below:
Now you know how the face landmark detection algorithm works. Now let’s implement the algorithm. For implementing that, you can see the code below along with explanations on each line of code:
By combining all the code as one, now let’s try the code! If the code doesn’t have any errors, the webcam will display the result along with the keypoints. In my case, here is the result:
Mediapipe is a tool for implementing ML-based computer vision solutions. The tool is created by Google.
This tool contains varieties computer vision solutions, such as face detection, pose estimation, object detection, and many more.
The advantage of this library is that you can apply the solutions on many platforms, such as web, mobile, PC, and many more.
I’ve already explained in the previous section to you how to implement face landmark detection using dlib. Now let’s implement the face landmark detection using Mediapipe.
The library uses the BlazeFace model for detecting face landmarks. BlazeFace is a deep learning model that is already optimized for low spec devices like smartphones. Therefore, we can use the model in real-time.
BlazeFace contains two main steps. First, the model detects one or more faces on an image. Second, the image detects around 468 face keypoints by using regression.
Different from the dlib library, this model detects 3D coordinates. Those x and y coordinates are normalized from the image scale. The z coordinate is retrieved by taking the relative calculation between the screen and the model x coordinates. You can read more details here.
Here is the flattened mesh from a face with their corresponding indexes:
In general, the pipeline for implementing face landmark detection is the same as the dlib library. It starts from importing libraries, initializing objects, detect face and its landmarks, and done.
Here is the code for doing that:
If you implement the code correctly, the image will display on your computer. Here is the preview of my result:
We have already take a walkthrough of face landmark detection libraries using dlib and mediapipe. We can say that both libraries are easy to use. Therefore, we can build our solution rapidly.
However, there are differences between them. The dlib library needs C++ dependencies it. That’s why we need CMake and Visual Studio for installing the library.
Also, this library needs a specific python library. Therefore, you have to create a virtual environment if you don’t have the supported Python version to run the library.
On the other side, installing mediapipe is easier. All you need to do is to install from pip only. Therefore, you don’t have to worry about installation more while using the mediapipe.
In the case of the solution, the dlib can detect only the 2D coordinates of the keypoints. On the other hand, the mediapipe can detect the 3D coordinates of the keypoints. Therefore, you can use those keypoints from the mediapipe library for estimating the head pose.
Well done! Now you know how to implement face landmark detection using Python. I’ve shown you the libraries like dlib and mediapipe for implementing the solution.
I hope it helps you in implementing a computer vision solution. Also, I hope it can become your foundation to build more complex applications.
If you are interested in my articles, you can follow me on Medium for more articles like this. Also, if you have any questions, you can contact me on LinkedIn.
Thank you for reading my article!
[1] https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/[2] https://www.analyticsvidhya.com/blog/2021/07/facial-landmark-detection-simplified-with-opencv/