Title: Facial landmarks with dlib, OpenCV, and Python - PyImageSearch

URL Source: https://pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/

Published Time: 2017-04-03T14:00:42+00:00

Markdown Content:
Last updated on July 3, 2021.

Last week we learned [how to install and configure dlib](https://pyimagesearch.com/2017/03/27/how-to-install-dlib/) on our system with Python bindings.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20568'%3E%3C/svg%3E)

Today we are going to use dlib and OpenCV to detect **_facial landmarks_** in an image.

Facial landmarks are used to localize and represent salient regions of the face, such as:

*   Eyes
*   Eyebrows
*   Nose
*   Mouth
*   Jawline

Facial landmarks have been successfully applied to face alignment, head pose estimation, face swapping, blink detection and much more.

In today’s blog post we’ll be focusing on the **basics of facial landmarks**, including:

1.  Exactly _what_ facial landmarks are and _how_ they work.
2.  How to detect and extract facial landmarks from an image using dlib, OpenCV, and Python.

In the next blog post in this series we’ll take a deeper dive into facial landmarks and learn how to extract _specific_ facial regions based on these facial landmarks.

**To learn more about facial landmarks, _just keep reading_.**

*   **Update July 2021:** Added section on alternative facial landmark detectors, including dlib’s 5-point facial landmark detector, OpenCV’s built-in facial landmark detector, and MediaPipe’s face mesh detector.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

The first part of this blog post will discuss facial landmarks and why they are used in computer vision applications.

From there, I’ll demonstrate how to detect and extract facial landmarks using dlib, OpenCV, and Python.

Finally, we’ll look at some results of applying facial landmark detection to images.

### What are facial landmarks?

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20578%20575'%3E%3C/svg%3E)

**Figure 1:** Facial landmarks are used to label and identify key facial attributes in an image ([source](http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html)).

Detecting facial landmarks is a _subset_ of the _shape prediction_ problem. Given an input image (and normally an ROI that specifies the object of interest), a shape predictor attempts to localize key points of interest along the shape.

In the context of facial landmarks, our goal is detect important facial structures on the face using shape prediction methods.

Detecting facial landmarks is therefore a two step process:

*   **Step #1:** Localize the face in the image.
*   **Step #2:** Detect the key facial structures on the face ROI.

Face detection (Step #1) can be achieved in a number of ways.

We could use OpenCV’s built-in Haar cascades.

We might apply a pre-trained [HOG + Linear SVM object detector](https://pyimagesearch.com/2014/11/10/histogram-oriented-gradients-object-detection/) specifically for the task of face detection.

Or we might even use deep learning-based algorithms for face localization.

In either case, the actual algorithm used to detect the face in the image doesn’t matter. Instead, what’s important is that through some method we obtain the face bounding box (i.e., the _(x, y)_\-coordinates of the face in the image).

Given the face region we can then apply **Step #2: detecting key facial structures in the face region.**

There are a variety of facial landmark detectors, but all methods essentially try to localize and label the following facial regions:

*   Mouth
*   Right eyebrow
*   Left eyebrow
*   Right eye
*   Left eye
*   Nose
*   Jaw

The facial landmark detector included in the dlib library is an implementation of the [_One Millisecond Face Alignment with an Ensemble of Regression Trees_](https://pdfs.semanticscholar.org/d78b/6a5b0dcaa81b1faea5fb0000045a62513567.pdf) paper by Kazemi and Sullivan (2014).

This method starts by using:

1.  A training set of labeled facial landmarks on an image. These images are _manually labeled_, specifying **specific** _(x, y)_\-coordinates of regions surrounding each facial structure.
2.  _Priors_, of more specifically, the _probability on distance_ between pairs of input pixels.

Given this training data, an ensemble of regression trees are trained to estimate the facial landmark positions directly from the _pixel intensities themselves_ (i.e., no “feature extraction” is taking place).

The end result is a facial landmark detector that can be used to **detect facial landmarks in _real-time_** with **high quality predictions**.

For more information and details on this specific technique, be sure to read the paper by Kazemi and Sullivan linked to above, along with the [official dlib announcement](http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html).

### Understanding dlib’s facial landmark detector

The pre-trained facial landmark detector inside the dlib library is used to estimate the location of _**68 (x, y)-coordinates**_ that map to facial structures on the face.

The indexes of the 68 coordinates can be visualized on the image below:

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201080%20870'%3E%3C/svg%3E)

**Figure 2:** Visualizing the 68 facial landmark coordinates from the iBUG 300-W dataset ([higher resolution](https://pyimagesearch.com/wp-content/uploads/2017/04/facial_landmarks_68markup.jpg)).

These annotations are part of the 68 point [iBUG 300-W dataset](https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/) which the dlib facial landmark predictor was trained on.

It’s important to note that other flavors of facial landmark detectors exist, including the 194 point model that can be trained on the [HELEN dataset](http://www.ifp.illinois.edu/~vuongle2/helen/).

Regardless of which dataset is used, the same dlib framework can be leveraged to train a shape predictor on the input training data — this is useful if you would like to train facial landmark detectors or custom shape predictors of your own.

In the remaining of this blog post I’ll demonstrate how to detect these facial landmarks in images.

Future blog posts in this series will use these facial landmarks to extract _specific_ regions of the face, apply face alignment, and even build a blink detection system.

### Detecting facial landmarks with dlib, OpenCV, and Python

In order to prepare for this series of blog posts on facial landmarks, I’ve added a few convenience functions to my [imutils library](https://github.com/jrosebr1/imutils), specifically [inside face\_utils.py](https://github.com/jrosebr1/imutils/tree/master/imutils/face_utils).

We’ll be reviewing two of these functions inside `face_utils.py` now and the remaining ones next week.

The first utility function is `rect_to_bb` , short for “rectangle to bounding box”:

def rect\_to\_bb(rect):
	# take a bounding predicted by dlib and convert it
	# to the format (x, y, w, h) as we would normally do
	# with OpenCV
	x = rect.left()
	y = rect.top()
	w = rect.right() - x
	h = rect.bottom() - y

	# return a tuple of (x, y, w, h)
	return (x, y, w, h)

This function accepts a single argument, `rect`, which is assumed to be a bounding box rectangle produced by a dlib detector (i.e., the face detector).

The `rect` object includes the _(x, y)_\-coordinates of the detection.

However, in OpenCV, we normally think of a bounding box in terms of _“(x, y, width, height)”_ so as a matter of convenience, the `rect_to_bb` function takes this `rect` object and transforms it into a 4-tuple of coordinates.

Again, this is simply a matter of conveinence and taste.

Secondly, we have the `shape_to_np` function:

def shape\_to\_np(shape, dtype="int"):
	# initialize the list of (x, y)-coordinates
	coords = np.zeros((68, 2), dtype=dtype)

	# loop over the 68 facial landmarks and convert them
	# to a 2-tuple of (x, y)-coordinates
	for i in range(0, 68):
		coords\[i\] = (shape.part(i).x, shape.part(i).y)

	# return the list of (x, y)-coordinates
	return coords

The dlib face landmark detector will return a `shape` object containing the 68 _(x, y)_\-coordinates of the facial landmark regions.

Using the `shape_to_np` function, we cam convert this object to a NumPy array, allowing it to “play nicer” with our Python code.

Given these two helper functions, we are now ready to detect facial landmarks in images.

Open up a new file, name it `facial_landmarks.py` , and insert the following code:

\# import the necessary packages
from imutils import face\_utils
import numpy as np
import argparse
import imutils
import dlib
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-p", "--shape-predictor", required=True,
	help="path to facial landmark predictor")
ap.add\_argument("-i", "--image", required=True,
	help="path to input image")
args = vars(ap.parse\_args())

**Lines 2-7** import our required Python packages.

We’ll be using the `face_utils` submodule of `imutils` to access our helper functions detailed above.

We’ll then import `dlib` . If you don’t already have dlib installed on your system, please [follow the instructions in my previous blog post](https://pyimagesearch.com/2017/03/27/how-to-install-dlib/) to get your system properly configured.

**Lines 10-15** parse our command line arguments:

*   `--shape-predictor` : This is the path to dlib’s pre-trained facial landmark detector. You can download the detector model [here](http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2) or you can use the **_“Downloads”_** section of this post to grab the code + example images + pre-trained detector as well.
*   `--image` : The path to the input image that we want to detect facial landmarks on.

Now that our imports and command line arguments are taken care of, let’s initialize dlib’s face detector and facial landmark predictor:

\# initialize dlib's face detector (HOG-based) and then create
# the facial landmark predictor
detector = dlib.get\_frontal\_face\_detector()
predictor = dlib.shape\_predictor(args\["shape\_predictor"\])

**Line 19** initializes dlib’s pre-trained face detector based on a modification to the standard [Histogram of Oriented Gradients + Linear SVM method](https://pyimagesearch.com/2014/11/10/histogram-oriented-gradients-object-detection/) for object detection.

**Line 20** then loads the facial landmark predictor using the path to the supplied `--shape-predictor`.

But before we can actually detect facial landmarks, we first need to detect the face in our input image:

\# load the input image, resize it, and convert it to grayscale
image = cv2.imread(args\["image"\])
image = imutils.resize(image, width=500)
gray = cv2.cvtColor(image, cv2.COLOR\_BGR2GRAY)

# detect faces in the grayscale image
rects = detector(gray, 1)

**Line 23** loads our input image from disk via OpenCV, then pre-processes the image by resizing to have a width of 500 pixels and converting it to grayscale (**Lines 24 and 25**).

**Line 28** handles detecting the bounding box of faces in our image.

The first parameter to the `detector` is our grayscale image (although this method can work with color images as well).

The second parameter is the number of image pyramid layers to apply when upscaling the image prior to applying the detector (this it the equivalent of computing [cv2.pyrUp](http://docs.opencv.org/3.1.0/d4/d86/group__imgproc__filter.html#gada75b59bdaaca411ed6fee10085eb784) _N_ number of times on the image).

The benefit of increasing the resolution of the input image prior to face detection is that it may allow us to detect _more_ faces in the image — the downside is that the larger the input image, the more computaitonally expensive the detection process is.

Given the _(x, y)_\-coordinates of the faces in the image, we can now apply facial landmark detection to each of the face regions:

\# loop over the face detections
for (i, rect) in enumerate(rects):
	# determine the facial landmarks for the face region, then
	# convert the facial landmark (x, y)-coordinates to a NumPy
	# array
	shape = predictor(gray, rect)
	shape = face\_utils.shape\_to\_np(shape)

	# convert dlib's rectangle to a OpenCV-style bounding box
	# \[i.e., (x, y, w, h)\], then draw the face bounding box
	(x, y, w, h) = face\_utils.rect\_to\_bb(rect)
	cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)

	# show the face number
	cv2.putText(image, "Face #{}".format(i + 1), (x - 10, y - 10),
		cv2.FONT\_HERSHEY\_SIMPLEX, 0.5, (0, 255, 0), 2)

	# loop over the (x, y)-coordinates for the facial landmarks
	# and draw them on the image
	for (x, y) in shape:
		cv2.circle(image, (x, y), 1, (0, 0, 255), -1)

# show the output image with the face detections + facial landmarks
cv2.imshow("Output", image)
cv2.waitKey(0)

We start looping over each of the face detections on **Line 31**.

For each of the face detections, we apply facial landmark detection on **Line 35**, giving us the 68 _(x, y)_\-coordinates that map to the specific facial features in the image.

**Line 36** then converts the dlib `shape` object to a NumPy array with shape _(68, 2)_.

**Lines 40 and 41** draw the bounding box surrounding the detected face on the `image` while **Lines 44 and 45** draw the index of the face.

Finally, **Lines 49 and 50** loop over the detected facial landmarks and draw each of them individually.

**Lines 53 and 54** simply display the output `image` to our screen.

### Facial landmark visualizations

Before we test our facial landmark detector, make sure you have upgraded to the latest version of `imutils` which includes the `face_utils.py` file:

$ pip install --upgrade imutils

_**Note:** If you are using Python virtual environments, make sure you upgrade the `imutils` inside the virtual environment._

From there, use the **_“Downloads”_** section of this guide to download the source code, example images, and pre-trained dlib facial landmark detector.

Once you’ve downloaded the .zip archive, unzip it, change directory to `facial-landmarks` , and execute the following command:

$ python facial\_landmarks.py --shape-predictor shape\_predictor\_68\_face\_landmarks.dat \\
	--image images/example\_01.jpg

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20568'%3E%3C/svg%3E)

**Figure 3:** Applying facial landmark detection using dlib, OpenCV, and Python.

Notice how the bounding box of my face is drawn in _green_ while each of the individual facial landmarks are drawn in _red_.

The same is true for this second example image:

$ python facial\_landmarks.py --shape-predictor shape\_predictor\_68\_face\_landmarks.dat \\
	--image images/example\_02.jpg

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20459%20600'%3E%3C/svg%3E)

**Figure 4:** Facial landmarks with dlib.

Here we can clearly see that the red circles map to specific facial features, including my jawline, mouth, nose, eyes, and eyebrows.

Let’s take a look at one final example, this time with multiple people in the image:

$ python facial\_landmarks.py --shape-predictor shape\_predictor\_68\_face\_landmarks.dat \\
	--image images/example\_03.jpg

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20459%20600'%3E%3C/svg%3E)

**Figure 5:** Detecting facial landmarks for multiple people in an image.

For both people in the image (myself and Trisha, my fiancée), our faces are not only _detected_ but also _annotated_ via facial landmarks as well.

### **Alternative facial landmark detectors**

Dlib’s 68-point facial landmark detector tends to be the most popular facial landmark detector in the computer vision field due to the speed and reliability of the dlib library.

However, other facial landmark detection models exist.

To start, dlib provides an [alternative 5-point facial landmark detector](https://pyimagesearch.com/2018/04/02/faster-facial-landmark-detector-with-dlib/) that is faster than the 68-point variant. This model works great if all you need are the locations of the eyes and the nose.

One of the most popular new facial landmark detectors comes from the [MediaPipe library](https://mediapipe.dev/) which is capable of computing a 3D face mesh:

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20400'%3E%3C/svg%3E)

**Figure 6:** Computing a 3D face mesh using the MediaPipe library ([image source](https://mediapipe.dev/)).

I’ll be doing tutorials on the PyImageSearch blog using MediaPipe and face mesh in the near future.

If you want to avoid using libraries other than OpenCV entirely (i.e., no dlib, MediaPipe, etc.), then it’s worth noting that [OpenCV _does_ support a built-in facial landmark detector](https://learnopencv.com/facemark-facial-landmark-detection-using-opencv/); however, I have not used it before and I cannot comment on its accuracy, ease of use, etc.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In today’s blog post we learned what facial landmarks are and how to detect them using dlib, OpenCV, and Python.

Detecting facial landmarks in an image is a two step process:

1.  First we must localize a face(s) in an image. This can be accomplished using a number of different techniques, but normally involve either Haar cascades or HOG + Linear SVM detectors (but any approach that produces a bounding box around the face will suffice).
2.  Apply the shape predictor, specifically a facial landmark detector, to obtain the _(x, y)_\-coordinates of the face regions in the face ROI.

Given these facial landmarks we can apply a number of computer vision techniques, including:

*   Face part extraction (i.e., nose, eyes, mouth, jawline, etc.)
*   Facial alignment
*   Head pose estimation
*   Face swapping
*   Blink detection
*   _…and much more!_

In next week’s blog post I’ll be demonstrating how to access each of the face parts _individually_ and extract the eyes, eyebrows, nose, mouth, and jawline features simply by using a bit of NumPy array slicing magic.

**To be notified when this next blog post goes live, _be sure to enter your email address in the form below!_**

![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!