Optimizing Face Detection on your browser with Tensorflow.js | by Siddhant Baldota | Towards Data Science

Let’s jump on to the most important question.
You might have come across a lot of face detection tutorials and projects. This one is different though. It is deployed on the browser and can be used anywhere from proctoring online exams to many different arcade games to using it for detecting face masks, blurring or improving the resolution of the face in real-time (since the region of interest is obtained) while also performing at quite high speeds. Machine learning on the web is the requirement of this age and shouldn’t be restricted to local systems. An alternative approach is face-api.js which uses the Multi-Task CNN but our detector has higher accuracy. Many other projects would be based on face detection models deployed on Flask Apps which are quite slow in comparison.
Tensorflow serving makes use of gRPC and Protobuf while Flask apps use REST and JSON. JSON relies on HTTP 1.1 while gRPC uses HTTP/2. HTTP 1.1 suffers from latency issues. A TCP handshake is needed for each individual request, and a large number of requests takes a significant toll on the time needed to load a page. HTTP 1.1 also suffers from Head of Line Blocking which restricts the number of connections to the same domain. With HTTP pipelining, you can send a request while waiting for the response to a previous request, effectively creating a queue. But that introduces other problems. If your request gets stuck behind a slow request then your response time will suffer. HTTP /2 maintains the basic premises and paradigms of HTTP but does away with optional parts of HTTP 1.1. REST supports only the request-response model available in HTTP 1.x. But gRPC takes full advantage of the capabilities of HTTP/2 and lets you stream information constantly. Protobuf is a binary format used to serialize data and is more efficient than JSON. Tensorflow serving can batch requests to the same model, which uses hardware (GPUs) more appropriately. Tensorflow serving has performance equal to code written in C/C++.
Moreover, Flask apps are written in Python whereas Tensorflow.js has Node.js or Chrome V8 engine as its server side node. In this project, we use Chrome’s V8 Javascript engine which is an open source high-performance JavaScript engine.
To get more intuition on the Chrome V8 engine, read this awesome blog at
www.freecodecamp.org
Tensorflow.js provides the right kind of hardware acceleration needed using the Web GL layer, which might be considered as a browser-side GPU. WebGL is a JavaScript API for rendering interactive 2D and 3D graphics within any compatible web browser without the use of plug-ins. WebGL is fully integrated with other web standards, allowing GPU-accelerated usage of physics and image processing and effects as part of the web page canvas.
Thus this client side serving face-detector proves to be quicker than the erstwhile flask face detection apps.
Tensorflow.js is a machine learning library in Javascript. It is an open-source hardware-accelerated JavaScript library for training and deploying machine learning models.
It can be used to develop ML in the browser by using flexible and intuitive APIs to build models from scratch using the low-level JavaScript linear algebra library or the high-level layers API. It can also be used to develop ML in Node.js by running native TensorFlow with the same TensorFlow.js API under the Node.js runtime. Pretrained Tensorflow or Keras models can be used in the browser by the TensorFlow.js model converters. Tensorflow.js can also retrain pre-existing model using sensor data-connected to the browser.
To know more about Tensorflow.js , you can check out the official documenation.
www.tensorflow.org
To get pretrained models, you can clone the following official tfjs GitHub repository
github.com
A text editor (optional) (Sublime, Brackets, Visual Studio Code, etc.)
You can download the Brackets editor from
brackets.io
Google Chrome
www.google.com
To use the browser based face detector, check out my GitHub repository
github.co
If you want to get hands on and do things on your own, here’s a guide
Create a starter HTML file
Add the following lines to the html file tensorflow.js headers to import the tfjs model in the head tag
<script src=”https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"></script><script src=”https://cdn.jsdelivr.net/npm/@tensorflow-models/blazeface"></script>
Styling (optional): Create the div(s) for formatting and add the style-sheet link. Add the video element and the canvas
Add the JavaScript file link in which the model is loaded
Rename your file to index.html
Your index.html file should look like this
Create a main.css file for some optional styling. Add the following code to it. Change as per convenience
You can get some lovely CSS at
codepen.io
Note: The main.css load a GIF which is for intial loading and is included in the repository. Feel free to use your own GIFs.
Create a blaze_pred.js file. Add the following code to it.
Explanation
We create a self calling function in which we obtain the canvas and video elements by their id. To get the context for the canvas, we use the getContext() method.
To recieve the video feed using the navigator.getUserMedia method and adding the stream to the video source object.
video.play() is used to play the video but since we have set the visibility to false in the video element of the index.html file, the video feed cannot be physically viewed. The feed which is displayed is the canvas. The face predictions are drawn using the canvas context.
To synchronize the video feed and the canvas element, we add an event listener and call a function draw which draws the predictions to it.
Create an asynchronous draw function with parameters as the video feed, context, the width and height of the canvas.
Draw the current frame of the video feed on it. Load the model using await blazeface.load(). Get the predictions on the model await model.estimateFaces() while setting the returnTensors parameter to false. Draw the bounding box on the canvas context using the predictions tensor. Obtain the confidence score using the predictions tensor and add the text to the canvas context relative to the bounding box. To call the draw function repeatedly set the timeout as 250 ms.
Add the working files (index.html, main.css, blaze_pred.js) to a folder. Name the folder tfjs-face_detection. You can give any other name too.
If you haven’t done the replication steps, type the following in your shell/terminal/command prompt
git clone https://github.com/sid0312/tfjs-face_detection
For Linux and Mac Users,
user@username:~$ cd tfjs-face_detectionuser@username:~/tfjs-face_detection$ google-chrome index.html
For Windows Users,
C:\Users\username> cd tfjs-face_detectionC:\Users\username\tfjs-face_detection> index.html
Allow access to the webcam
Wait for a few seconds and you’re all set!
We have successfully detected our face on the browser using Tensorflow.js. You can create your own deep learning face model and convert it into a tfjs model using the tfjs model converter to increase its robustness.
Yeah! The one in the video feed is me! Check out my other articles:
towardsdatascience.com
towardsdatascience.com
Happy Computer Vision, Happy Deep Learning and Happy Deployment. Until next time!