Face detection using Face-Api.js and Flask | by Karthikraj Naidu | Towards Data Science

There are numerous face detection systems available online for Python like Dlib, OpenCV, and other Object Detection Systems by Deep Learning. We may have already used OpenCV to use a frame for capturing video from webcam and doing facial landmark detection using Dlib, MTCNN, etc. But I always got fed up that in a project I wasn’t able to add it in the UI or there are too many frame skips due to the load on CPU.
Most of the people use Flask as the goto framework for deploying their ML/DL model and beauty of creating UI from HTML and managing backend from python server works like a charm. But whenever you try to include webcam face detection for your web app you get stuck with a dialog box from Opencv imshow() and bust up the UX. The development of TF.js has empowered us to develop complex models on the browser itself. But python freaks like us with minimal knowledge on Node.js and other web development terminologies had to know on how to use these tools in our Flask playground
In this article, we would be creating a Flask web app with face recognition integrating into the browser
Advantages
Embed the video stream of data and inference into browser content
Less Utilization of Resources and improved FPS
No need to shift projects from Python to Node.js
As I was looking by a solution for face detection for Flask I came across a JavaScript API for face detection and face recognition in the browser made by Vincent Mühler who has done awesome work on creating APIs for TF.js implementation of Face detection and Recognization. These API can be used to extract data from the Webcam video and also to draw the landmarks on the Video frames itself.
github.com
Our project requires the following dependencies to be installed
Flask
pip install Flaskconda install -c anaconda flask
Flask-SocketIO
pip install Flask-SocketIOconda install -c conda-forge flask-socketio
Our Flask app creation consists of 3 parts
Step 1: Creating a static webpage using Face-Api.js for landmark detection
Step 2: Adding the webpage in Flask Server and Passing inferences from the webpage asynchronously to the flask backend to process
In this step, we will be using Face-Api.js to create static webpages for face detection and logging the inferences on the console.
The File Structure which I am using is as follows
|--static|     |--js|     |     |--face-api.min.js|     |     |--index.js|     |--styles|     |     |--style.js|     |--models|     |     |-- // All the models from Face-Api.js|--template|     |--index.html|-app.py
The latest face-api.min.js is available to download here and the weights to be included in the models’ folder can be downloaded here
We will create a basic HTML file with a video tag to show the output and add all the script and CSS files to the document in template/index.html
To improve the basic UI CSS is added in static/styles/style.css
Here comes the Magic... using Face-Api.js for face landmark detection. All the code is added in the static/js/index.js
The above code is used to load the models and initialize the Face-api.js
Then we would access the webcam and start to stream video data
The startVideo() function is invoked once the face API is loaded
After the video data is loaded we would add listeners to the video stream and add face-api.js inferences
That’s it .... now you can view the webpage by using VS Code Live Server or by Web Server for Chrome ...select the folder and start the server and enter the URL in chrome tab
Yay !! we have completed the first step of using face-api.js to do face landmark detection
But still, we aren’t able to get this data in python. So let’s head to step 2...
Let’s work on creating a basic flask app to view the webpage. Create a basic Flask app to render the webpage we created at app.py
This might view the webpage but still, there is no way there is contact between the webpage and flask app to send and receive inference data. So we would be using Sockets for continuous data transfer
Enter SocketIO
Socket IO is useful to create client-server sockets between webpage and servers. The complexities of connection and sending and receiving data are carried by SocketIO functions so that we can focus on the main processing of data
socket.io
Creating a web socket in Flask App
After installing the Flask-SocketIO dependencies we can update the app.py as per the following
The above code creates a server-side socket for our App. So we need to create a client-side socket for our web app
As you might know, connecting static files in Flask uses url_for(). Adding SocketIO dependencies to index.html and to link other static files in folders update the code accordingly
To create a socket on Javascript side first initialize socket and test connection
Due to the creation of the flask server, the face-api.js is not able to directly locate model weights as we had initialized before so need to update the models’ folder location in index.js
To send the Data using socket add the following piece inside the setInterval() This will constantly send a stream of data to Flask.
That’s All...Your data from the Browser will be sent to the flask Backend and will be printed on the terminal. you can do the post-processing accordingly
To sum up the code implementation
Demo of Face Detection using Flask
I hope this was an interesting read for you guys. Thanks to thirumalainambi Yadav for support on the JS part of the project.
Face-Api.js by Vincent Mühler from https://justadudewhohacks.github.io/face-api.js/docs/globals.html
SocketIO example programs from here
The entire project code is available in the following Github Repository
github.com
Thanks for having a read
For Contacting
Drop me a mail at karthikraj.v17@siesgst.ac.in
Or connect me through the following links