Face landmarks detection with MediaPipe Facemesh | by Benson Ruan | Towards Data Science

Tensorflow.js released the MediaPipe Facemesh model in March, it is a lightweight machine learning pipeline predicting 486 3D facial landmarks to infer the approximate surface geometry of a human face.
During the pandemic time, I stay at home and play with this facemesh model. I would like to remind people of the importance of wearing a face mask. So I built a virtual face mask fitting room, which can detect face landmarks in an image or webcam stream, and put on the selected face mask on your face.
Try it yourself in the link below :
bensonruan.com
Wearing a face mask is a gesture of concern for one’s community. It prevent the wearer from inadvertently passing the disease along to others. The virtual face mask fitting room utilizes an advanced technique called Face Landmark Detection, which can identify human faces in an image or video stream.
If you are interested in building a face landmark detection app, please follow me below for the journey of how I implemented it.
First of all, simply include the script Tensorflow.js and its facemesh model in the <head> section of the html file.
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script><script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script><script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/facemesh"></script>
Or you can install it via npm for use in a TypeScript / ES6 project
npm install @tensorflow-models/facemesh//import in jsconst facemesh = require('@tensorflow-models/facemesh');
The next thing we will need to do is to add the html <img> or <video> element as the source, so that we can perform face landmark detection on that image or webcam video stream.
<img id="faces" src="images/Donal_Thrump_White_House.jpg"><video id="webcam" autoplay playsinline width="640" height="480"></video>
To stream your webcam into the browser, I utilize the npm JavaScript module webcam-easy.js, which provides an easy to use module that can access webcam and take a photo. To find out more details about that, please refer to my previous blog :
medium.com
In order to perform face landmark detection, we first need to load the pre-trained Facemesh model, by calling the API of facemesh.load(modelParams). FaceMesh comes with a few optional parameters of the model:
maxContinuousChecks (default value : 5)
— How many frames to go without running the bounding box detector. Only relevant if maxFaces > 1
detectionConfidence (default value : 0.9)
— Threshold for discarding a prediction
maxFaces (default value : 10)
— The maximum number of faces detected in the input. Should be set to the minimum number for performance
iouThreshold (default value : 0.3)
— A float representing the threshold for deciding whether boxes overlap too much in non-maximum suppression. Must be between [0, 1]
scoreThreshold (default value : 0.75)
— A threshold for deciding when to remove boxes based on score in non-maximum suppression
facemesh.load().then(mdl => {     model = mdl;    console.log("model loaded");    cameraFrame =  detectFaces();});
Next, we start to feed the image or webcam stream through the Facemesh model to perform face landmark detection, by calling the API of model.estimateFaces(inputElement). It takes an input image element (Can be a tensor, DOM element image, video, or canvas) and returns an array of facial landmark key points, bounding boxes and confidence level.
The estimateFaces API comes with a few parameters of the model:
input
— The image to classify. Can be a tensor, DOM element image, video, or canvas.
returnTensors (default value : false)
— Whether to return tensors as opposed to values
flipHorizontal (default value : false)
— Whether to flip/mirror the facial keypoints horizontally. Should be true for user facing webcam stream
let inputElement = isVideo? webcamElement : imageElement;let flipHorizontal = isVideo;model.estimateFaces(inputElement, false, flipHorizontal).then(predictions => {    console.log(predictions);    drawMask(predictions);});
Return of predictions would look like:
For each of the facial landmark key point, it contains the location of the x, y axis and depth.
0: Array(3)    0: 202.05661010742188    1: 207.98629760742188    2: -5.985757827758789    length: 3
In the above function, we get 468 face landmark key points. For our face mask application, I utilize 4 of those landmarks:
Forehead: 10
Left Cheek: 234
Chin: 152
Right Cheek: 454
Then we can use those key points to calculate where we should overlay the face mask PNG image. We need to calculate 3 numbers below:
The (x,y) location of the top-left corner for the mask image
The width of the mask image
The height of the mask image
Here is how I get the mask image with: use the Right Cheek landmark’s x axis value — Left Cheek landmark’s x axis value
maskWidth =(dots[rightCheekIndex].left - dots[leftCheekIndex].left);
For the height and top-left corner location, there is a difference between Half mask and Full mask.
Full Mask
— Top-left corner: {x: Forehead x axis value; y: Left Cheek y axis value}
— Height: [Chin y axis value] — [Forehead y axis value]
Half Mask
— Top-left corner: {x, y : Left landmark’s x,y axis value}
— Height: [Chin y axis value] — [Left Cheek y axis value]
switch(maskType) {  case 'full':    maskCoordinate= { top: dots[foreheadIndex].top, left: dots[leftCheekIndex].left};    maskHeight = (dots[chinIndex].top - dots[foreheadIndex].top) ;    break;  case 'half':    maskCoordinate = dots[leftCheekIndex];    maskHeight = (dots[chinIndex].top - dots[leftCheekIndex].top) ;    break;}
The last step is to overlay the face mask PNG image to the face.
maskElement = $("<img src='"+selectedMask.attr('src')+"' class='mask' />");maskElement.appendTo($("#canvas"));maskElement.css({    top: maskCoordinate.top,     left: maskCoordinate.left,     width: maskWidth ,    height: maskHeight,    position:'absolute'});
That’s pretty much for the code! Now choose your favorite mask and try it on yourself!
You can download the complete code of the above demo in the link below:
github.c
Facemesh model is designed for front-facing cameras on mobile devices, where faces in view tend to occupy a relatively large fraction of the canvas. MediaPipe Facemesh may struggle to identify far-away faces.
With fears of spreading the coronavirus at the forefront of everyone’s minds, facial detection & recognition systems may be a safer, cleaner option than traditional bio-metric access control systems. This technology can not only reduce the risk of cross-infection but also improve traffic efficiency by more than 10 times, which will save time and reduce congestion.
At last, I wish everyone to stay healthy and stay at home during the pandemic, let’s utilize the latest machine learning technology to help us fight against the virus.
Thank you for reading. If you like this article, please share on Facebook or Twitter. Let me know in the comment if you have any questions. Follow me on GitHub and Linkedin.