How to Deploy a Pre-trained Keras Model with OpenCV and Flask | by Jacob Tadesse | Towards Data Science

In this post, I will share how to deploy a pre-trained model to a locally hosted computer with Flask, OpenCV and Keras. I initially deployed this model on PythonAnywhere using Flask, Keras and jquery. The application was designed for remote school classroom or workplace settings that require students or employees to shave their facial hair.
The application allowed users to upload a photo, and click a button to send a post request with the encoded image data to the backend of the website. The image transformation and classification were handled on the backend, and the results were returned to the front-end in an html response. The response updated the page dynamically, as Shaved or Unshaved.
The backend of the application was built in Flask, but I wanted to allow a live video stream that would detect a user’s face, labeling the classification on screen. So since I already had a model and a basic Flask framework, I wanted to use OpenCV to do the rest on my local machine.
If you have a model, you can follow the same format, but if you do not, I would recommend reading my previous blog posts (Building a Convolutional Neural Network to Recognize Shaved vs UnShaved Faces & How to Split a Pickled Model File to Bypass Upload Limits on PythonAnywhere).
OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.
https://opencv.org/about/
After reading documentation, watching YouTube videos, and reading blog posts, I found a very helpful article, COVID-19: Face Mask Detection using TensorFlow and OpenCV by Gurucharan M K. While Gurucharan used OpenCV to detect Face Masks for COVID-19, I wanted to use a similar format to detect if a faces first, then determine if the face was shaved or not.
The most helpful piece to my puzzle was detecting the faces first, and this was accomplished by using the Face detection program the Haar Feature-based Cascade Classifiers for detecting the features of a face.
face_classifier = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
Haar Cascade is a machine learning object detection algorithm used to identify objects in an image or video and based on the concept of ​​ features proposed by Paul Viola and Michael Jones in their paper “Rapid Object Detection using a Boosted Cascade of Simple Features” in 2001
http://www.willberger.org/cascade-haar-explained/
Video Capture
video = cv2.VideoCapture(0)
Stop Video
video.release()
Display Video
(rval, im) =video.read()
Flip Video
im = cv2.flip(im, 1, 1)
Resize Video
mini = cv2.resize(im, (im.shape[1] // 4, im.shape[0] // 4))
Detect Faces
faces = face_classifier.detectMultiScale(mini)
Flask is a micro web framework written in Python. It is classified as a microframework because it does not require particular tools or libraries. It has no database abstraction layer, form validation, or any other components where pre-existing third-party libraries provide common functions.
Anmol Behl has a well-written article entitled, “Video Streaming Using Flask and OpenCV“. Following this article, I was able to build out an html webpage template, as well as the routes needed to stream video for each frame’s predictions to the browser using the computer’s built-in camera.
The html page was similar to the page I built before, without the jquery.
Here is the simple index.html code I used to display the homepage:
<!DOCTYPE html><html lang="en">  <head>    <title>Shaved or Not_Shaved </title></head>  </head>  <body>    <center><h1>Shaved or Not_Shaved Streaming App Demo</h1></center>    <center><img id="bg" src="{{ url_for('video_feed') }}"></center>    <video id="video" autoplay>Video Stream not available.</video>  </body></html>
To build out the routes, I used the following code to handle when the page was loaded, as well as the video camera images pulled from our camera.py file that is initialized when the website is loaded.
Here is the Flask code I used in the main.py file:
from flask import Flask, render_template, Responsefrom camera import VideoCameraimport osos.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'app = Flask(__name__)@app.route('/')def index():    # rendering webpage    return render_template('index.html')def gen(camera):    while True:        #get camera frame        frame = camera.get_frame()        yield (b'--frame\r\n'               b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n\r\n')@app.route('/video_feed')def video_feed():    return Response(gen(VideoCamera()),                    mimetype='multipart/x-mixed-replace; boundary=frame')if __name__ == '__main__':    # defining server ip address and port    app.run(host='0.0.0.0',port='5000', debug=True)
Keras is a deep learning API written in Python, running on top of the machine learning platform TensorFlow. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result as fast as possible is key to doing good research.
https://keras.io/about/
Since Keras runs on top of TensorFlow, we can utilize the Graph and Session methods from the TensorFlow module. While the detail of graph and session is outside of the scope of this article, you can still use the following code when deploying your model to be accessible during scheduled sessions.
import picklefrom keras import backend as Kfrom tensorflow import Graph, Sessionglobal loaded_modelgraph1 = Graph()with graph1.as_default():	session1 = Session(graph=graph1)	with session1.as_default():		loaded_model = pickle.load(open('Combined_Model.p', 'rb'))
The final piece is our camera.py file. In this example, our file will contain a function to join our split model into one pickle file. We will use a class VideoCamera to initiallize the camera when the app loads, and also combine the face detection program with our pre-trained Keras model.
import cv2import numpy as npimport picklefrom keras import backend as Kfrom tensorflow import Graph, Session# defining face detectorclassifier=cv2.CascadeClassifier('haarcascade_frontalface_default.xml')size = 4labels_dict={0:'shaved',1:'not_shaved'}color_dict={0:(0,255,0),1:(0,0,255)}global loaded_modelgraph1 = Graph()with graph1.as_default():	session1 = Session(graph=graph1)	with session1.as_default():		loaded_model = pickle.load(open('Combined_Model.p', 'rb'))class VideoCamera(object):    def __init__(self):        # capturing video        self.video = cv2.VideoCapture(0)    def __del__(self):        # releasing camera        self.video.release()    def get_frame(self):        # extracting frames        (rval, im) = self.video.read()        im = cv2.flip(im, 1, 1)        mini = cv2.resize(im, (im.shape[1] // size, im.shape[0] // size))        faces = classifier.detectMultiScale(mini)        for f in faces:            (x, y, w, h) = [v * size for v in f] #Scale the shapesize backup            #Save just the rectangle faces in SubRecFaces            face_img = im[y:y+h, x:x+w]            resized=cv2.resize(face_img,(300,300))            normalized=resized/255.0            reshaped=np.reshape(normalized,(1,300,300,3))            reshaped = np.vstack([reshaped])            K.set_session(session1)            with graph1.as_default():                results=loaded_model.predict(reshaped)            if results >.5:                result = np.array([[1]])            else:                result = np.array([[0]])            label = np.argmax(result)            cv2.rectangle(im,(x,y),(x+w,y+h),color_dict[result[label][0]],2)            cv2.rectangle(im,(x,y-40),(x+w,y),color_dict[result[label][0]],-1)            cv2.putText(im, labels_dict[result[label][0]], (x, y-10),cv2.FONT_HERSHEY_SIMPLEX,0.8,(255,255,255),2)            # encode OpenCV raw frame to jpg and displaying it        ret, jpeg = cv2.imencode('.jpg', im)        return jpeg.tobytes()
Please note that this application is limited to your local computer. I hope this article is helpful. I would love to deploy this model publicly using Node.js, Tensorflow.js and OpenCV.js, but this is outside the scope of this article. If I get 100 comments, I will make that my next post!
https://github.com/cousinskeeta/shavedVnotShaved_opencv
COVID-19: Face Mask Detection using TensorFlow and OpenCV by Gurucharan M K
Video Streaming Using Flask and OpenCV by Anmol Behl