How to Detect Mouth Open for Face Login | by Peter Xie | Towards Data Science

Have you ever used bank/payment app face login and been asked to open mouth, nod or turn head? Such methods have been very popular, especially in China, to prevent deceiving/hacking using static face images or 3D prints. I spent about two days and finally figured out a quite simple way to detect mouth open utilizing the feature outputs from the face_recognition project.
Here is a quick look at the effect when applying the algorithm to real-time webcam video.
face_recognition is an awesome open source project for face recognition based on dlib, just as described by itself:
The world’s simplest facial recognition api for Python and the command line
With a few lines of Python code, you can detect a face, recognize who it is and output face features, such as chin, eyes, nose, and lips.
For example, run the sample find_facial_features_in_picture.py (only 40 lines of code) for an Obama image, you can get all facial features as shown below.
You will also get the facial feature details in standard output, including top lip and bottom lip. Each feature is a list of positional points.
You can follow the steps in the face_recognition project on github, or just simply download the pre-configured VM (strongly recommended).
The face_recognition provides APIs for static figure analysis, but it won’t tell you facial motions like mouth opening or nod. However, we can detect these motions by the feature outputs, i.e., mouth open/close in this case.
Both top lip and bottom lip feature outputs contain a list of 12 positional points, but in different orders.
top_lip points: [(181, 359), (192, 339), (211, 332), (225, 336), (243, 333), (271, 342), (291, 364), (282, 363), (242, 346), (225, 347), (211, 345), (188, 358)]bottom_lip points: [(291, 364), (270, 389), (243, 401), (223, 403), (207, 399), (190, 383), (181, 359), (188, 358), (210, 377), (225, 381), (243, 380), (282, 363)]
Plot the lip points with matplotlib and you can see clearly the point order as shown below. Check out the plot script here if interested.
My algorithm is very simple:
If mouth open height is greater than lip height * ratio, mouth is open.
ratio: is adjustable and define how much the mouth is open. You can simply put ratio = 1, which means the mouth is open more than lip height.
lip height: Average of the distances of the three pairs of points as follows:
2–103–94–8
These pairs are the same for both top and bottom lip points.
mouth height: Average of the distances of the three pairs of points as follows:
Top lip 8–10 Bottom lipTop lip 9–9  Bottom lipTop lip 10–8 Bottom lip
lip height function:
If you add up the indexes of each point pairs, you will find the sum is always 12. So if i is one point, the other point is 12-i.
The distance of two points (x1,y1) and (x2,y2) is defined as follows.
Thus we get the lip height function below.
mouth height function:
Add up the indexes of each mouth point pairs, the sum is 18.
check mouth open function:
With lip height and mouth height functions, we can define the check mouth open function as below.
I choose the min value of top lip height and bottom lip height as lip height. Or you can use the average value.
And I pick a ratio as 0.5, so I don’t need to open a big mouth to test it 😄.
Okay, everything is clear and ready. Let’s test it out.
Outputs:
top_lip height: 12.35bottom_lip height: 21.76mouth height: 33.34Is mouth open: True
Combine the face_recognition and the mouth open detection algorithm, we can develop a webcam real-time face recognition app with the ability to detect mouth open/close as you see from the video in the beginning of this post. It is just 80 lines of code and you can clone from github here. Note that you need to install face_recognition package first.
git clone https://github.com/peterjpxie/detect_mouth_open.gitcd detect_mouth_openpython facerec_from_webcam_mouth_open.py
This is just one way to detect mouth open. You will definitely find other approaches, similar or not, from other sources, or you can figure out your own method. It is just fun and exciting to work out a method yourself before searching for an answer.
Thanks for reading.