Head Pose Estimation using Python | Towards Data Science

Head pose estimation is one of the computer vision tasks that exist. In this task, we want to know the object’s pose from its translation and rotation.
This task has lots of applications. For example, we can detect whether the driver is paying attention to the road or not. The second example is that we can check whether the student is getting distracted from study or not. And there are lots of applications that we can do with it.
As you know, we have only a two-dimensional image. How can we estimate the pose of an object by relying on the image itself? We can use a solution called the Perspective-n-Point (PnP).
The PnP problem equation looks like this:
From this equation, we can retrieve the rotational and the translational matrix. But before we get those matrices, this equation needs to take three inputs, such as:
The 2D coordinates in the image space
The 3D coordinates in the world space
The camera parameters, such as the focal points, the center coordinate, and the skew parameter
From that equation, we got two main problems:
How can we get those inputs?
How can we estimate the pose from an object based on those inputs?
This article will show you how. In this article, we will use Python as the programming language. Also, we use the mediapipe library for detecting face keypoints and the OpenCV library for estimating the pose of a head.
Here is the preview of what we will create:
Without further, let’s get started!
Before we get into the implementation, the first step is to install the libraries. In this case, we will install the OpenCV and the Mediapipe library by using pip. On your terminal, please write this command:
pip install opencv-pythonpip install mediapipe
After we install the libraries, the next step is to load the libraries into our code. We will import the NumPy, OpenCV, and Mediapipe libraries. Please add this line of code:
After we load the libraries, the next step is to initialize several objects. There are two objects that we initialize. They are:
The FaceMesh object from the Mediapipe library. This object will detect faces and also detect keypoints from one or more faces.
The VideoCapture object from the OpenCV library. This object will be used for retrieving images from the webcam. We set a parameter on the object with 0 for retrieving images from the webcam.
Please add these lines of code:
Now we have initialized the objects. The next step is to capture the image from the webcam. Please add this line of code for doing that:
After we capture the image, the next step is to process the image. For your information, the OpenCV and the Mediapipe library read their image differently.
On the OpenCV library, the image is in BGR color space. Meanwhile, the mediapipe library needs an image with RGB color space.
Therefore, we need to convert the color space to RGB first, apply face landmark detection, then convert it back to BGR color space.
Please add these lines of code (Be careful with the indentation):
After we process the image, the next step is to retrieve the keypoint coordinates. For your information, the mediapipe’s face landmark detection algorithm catches around 468 keypoints from a face. Each keypoint is on 3D coordinates.
For head pose estimation, we don’t have to use all the keypoints. Instead, we choose 6 points that at least can represent a face. Those points are on the edge of the eyes, the nose, the chin, and the edge of the mouth.
For accessing those points, we refer to the index that has been used on the BlazeFace model. I’ve already marked the index. Here is the picture of it:
Now let’s extract those keypoints. For 2D coordinates, we will take only the x and y-axis coordinates. And for 3D coordinates, we retrieve all of the axes.
But before we extract those keypoints, we have to multiply the x-axis with the image’s width. Also, we multiply the y-axis with the image’s height.
Also, we will take the nose coordinates. We do that to display the projection of our nose on the image space.
Now let’s add these lines of code, and be careful with the indentation:
Now we have the 2D and 3D coordinates of our face keypoints. The next step is to get the camera matrix. Let’s recall the camera matrix once again.
As you can see from above, we need to have several parameters. The first one is the focal point. We can get the focal point (fx and fy) by taking the width of the image.
The second parameter that we will take is the skew parameter. The parameter has the symbol of gamma. For this parameter, we set the value to 0.
The third parameter is the center coordinate of our image. We will set the u0 with the image’s width set the v0 with the image’s height.
Now let’s create the matrix by generating a NumPy array. Now let’s add these lines of code:
Before we apply the PnP problem, we need to add another matrix. It’s a distance matrix. This matrix only contains zero, and it has a shape of 4x1. Now let’s create the matrix by adding this line of code:
We have all the inputs, ranging from the 2D coordinates, the 3D coordinates, the camera parameters matrix, and the empty distance matrix. Let’s apply the PnP to our problem by adding this line of code:
From this process, now we have the translational vector and the rotational vector. Wait, the rotational section is not in a matrix format. We cannot retrieve the rotation angle with it.
Don’t worry. We can convert the vector into the matrix by using the cv2.Rodrigues function.
Now let’s add this line of code:
Now we have the rotational matrix. Now let’s retrieve the rotational angle on each axis. For doing that, we can use the cv2.RQDecomp3x3 function for extracting the angles.
Now let’s add this line of code:
The last step that we have to do now is to determine where our head is heading. We display the direction and create a line to see the projection of our nose on the image space.
Now let’s add these lines of code:
By combining all of those lines of code, the result will look like this:
Well done! We have implemented the head pose estimation using Python. I hope it helps you in building your computer vision solution, especially the head pose estimation problem.
If you are interested in my articles, you can follow me on Medium. If you have any questions, you can connect with me on LinkedIn.
In case that you have confusion on writing the code, you can check the complete code here to help you:
Thank you for reading my article!
[1] https://medium.com/analytics-vidhya/real-time-head-pose-estimation-with-opencv-and-dlib-e8dc10d62078[2] https://learnopencv.com/head-pose-estimation-using-opencv-and-dlib/