Detect Facial Landmark Points With C# And Dlib In Only 50 Lines Of Code
By: Mark Farragher
Tags: Dotnet Core, Computer Vision, Programming, Dotnet, Facedetection

A hot research area in computer vision is to build software that understands the human face. The most obvious application is Face Recognition, but we can also do lots of other cool stuff like Head Pose Estimation, Emotion Detection, Eye Gaze Detection, and Blink Detection.

The first step in building any face analysis app is to do Face Detection: scan an image and find all rectangular areas that contain a face.

Check out my other Medium article that covers face detection in detail.

But face detection is not enough. We need to really analyze all the details of a face and zoom into areas like the eyes, nose, and lips to have any chance of doing reliable emotion detection or head pose estimation.

For that, we need to detect Facial Landmark Points.

Landmark points are predetermined and numbered locations on a face:

So if I want to detect the tip of someone’s chin, I would detect the facial landmark points and then focus on point number 8, which tracks the tip of the chin in a face.

Likewise, if I want to track eyebrows, I would monitor landmark points 17–21 and 22–26 which track the left and right eyebrows.

Detecting emotions or tracking head pose is now possible, by monitoring the landmark points and analyze how they change over time.

I am going to build an app that can detect all landmark points in all faces that are visible in an image. I’ll use C#, Dlib, DlibDotNet, and NET Core v3, and try to achieve my goal with the minimum of code.

Dlib is the go-to library for face detection. It’s intended for C and C++ projects, but Takuya Takeuchi has created a NuGet package called DlibDotNet that exposes the complete Dlib API to C#.

And NET Core is the Microsoft multi-platform NET Framework that runs on Windows, OS/X, and Linux. It’s the future of cross-platform NET development.

The Dlib library has a built-in landmark detector that can recognize 68 landmark points on a face that cover the jaw, chin, eyebrows, nose, eyes, and lips.

I’ll use the following image to test my app:

This is the famous selfie Ellen DeGeneres took at the Oscars in 2014. It’s a great test image because everybody is looking at the camera and we have a couple of celebrities with their face only partly visible.

Save this image as ‘input.jpg’.

Let’s get started. Here’s how to set up a new console project in NET Core:

$ dotnet new console -o DetectLandmarks

$ cd DetectLandmarks

Next, I need to install the ML.NET packages I need:

$ dotnet add package DlibDotNet

That was easy! This single NuGet package installs Dlib and the DlibDotNet wrapper, and sets everything up for your operating system.

If you’re working on a Mac like me, you’ll have to perform one extra step. DlibDotNet requires the XQuartz library but it’s not installed by default on a clean OS/X system.

You can easily install XQuartz with homebrew:

$ brew cask install xquartz

And that’s it.

Now I’m ready to add some code. Here’s what Program.cs should look like:

The Dlib.GetFrontalFaceDetector method loads a face detector that’s optimized for frontal faces: people looking straight at the camera.

And ShapePredictor.Deserialize loads a 68-point configuration file for Dlib’s built-in landmark detector and initializes it. You can download this file here.

Dlib uses loadable configuration files which is really nice. This particular file uses 68 landmark points, but I can also load other files with many more points that track even more details in a face.

Next, I have to load the image and perform landmark detection:

The Dlib.LoadImage<RgbPixel> method loads the image in memory with interleaved color channels. The Operator method then performs face detection on the image.

This is important. I need to do face detection first, because the landmark detector will only work if I tell it which part of the image contains a face. If I ty to run the landmark detector directly on the entire image, it won’t work.

So now the faces variable holds an array of Rectangle structs. Each rectangle describes where the face detector found a face in the image.

For each rectangle, I fire up the landmark detector will a call to Detect. Note that I’m providing the image and the face rectangle, so now the detector knows where to look for landmark points.

In computer vision we always use this 2-step process: first detect the faces, and then detect the landmark points within the face rectangle.

The shape variable now contains the set of landmark points, and shape.Parts contains the number of points. I get every point using GetPart, convert it into a rectangle, and draw it directly on the image using Dlib.DrawRectangle.

The final step is to save the modified image:

This uses the Dlib.SaveJpeg method to save the image as output.jpg.

And that’s it!

You can run this app on Linux, OS/X or Windows with Visual Studio Code: