Step by Step Face Recognition Code Implementation From Scratch In Python | by Rohit Thakur | Towards Data Science

Face Recognition is a very popular topic. It has lot of use cases in the filed of biometric security. Now a days with the help of Deep learning face recognition has become very feasible to people. As deep learning is a very data intensive task and we may always not have such huge amount of data to work in case of face recognition so with the advancement in One Shot Learning, face recognition has become more practical and feasible. To make it even more feasible, simple and easy to use we have created a python package that is even more easy to use. We have eliminated all the steps to download the supporting files and setting up the supporting files. You can simply installed the python package and start doing face detection and recognition. The python package is explained in the later part of the article.
First in this article we will be going through all the steps to implement One shot Learning for Face Recognition in Python. The Face Recognition consists of 2 parts. They are :
Face Detection in the ImagePerforming Face Recognition on the detected image
Face Detection in the Image
Performing Face Recognition on the detected image
First Step is to download the dataset so that we can start to run the code. There are a lot of face dataset available online. I would be using Labeled Faces in the Wild dataset. You can download the dataset from the link below.
http://vis-www.cs.umass.edu/lfw/#download
Once you have downloaded the dataset then you can start with the code written below.
import dlib, cv2,osimport matplotlib.pyplot as pltimport numpy as npfrom imutils.face_utils import FaceAligner
Here we have imported some of the python packages. Dlib will be used for facial landmark detection. The pre-trained facial landmark detector inside the dlib library is used to estimate the location of 68 (x, y)-coordinates that map to facial structures on the face. Cv2 will be used to perform operations on the image. Imutils Face Aligner will be used to align the different orientations of faces so that all the faces are forward looking faces.
pose_predictor=dlib.shape_predictor(‘shape_predictor_68_face_landmarks.dat’)fa = FaceAligner(pose_predictor)face_encoder=dlib.face_recognition_model_v1(‘dlib_face_recognition_resnet_model_v1.dat’)detector = dlib.get_frontal_face_detector()modelFile = ‘opencv_face_detector_uint8.pb’configFile = ‘opencv_face_detector.pbtxt’net = cv2.dnn.readNetFromTensorflow(modelFile, configFile)
Here in this step we are loading all the pre trained packages. We will be needing these 4 pre trained files to work with. You can download these 4 files from the github repo linked below.
git clone https://github.com/1297rohit/FaceReco.git
pose_predictor is used to get the 68 facial landmarks detection.face_encoder is used to get the (128,1) dimension encoding of the image which is passed to it. It is a pretrained ResNet network model with 29 conv layers. The model is trained on a dataset of about 3 million faces.detector is an object of dlib.get_frontal_face_detector() which is used to get the front face from the face image.modelFile, configFile are the files for the dnn based cv2 face detection model. It is a quantized tensorflow model which is based on the Single Shot-Multibox Detector (SSD) and uses ResNet-10 architecture as its backbone. It was introduced post OpenCV 3.3 in its deep neural network module.net is the object of the cv2.dnn module that is initialized with the model and config file.
pose_predictor is used to get the 68 facial landmarks detection.
face_encoder is used to get the (128,1) dimension encoding of the image which is passed to it. It is a pretrained ResNet network model with 29 conv layers. The model is trained on a dataset of about 3 million faces.
detector is an object of dlib.get_frontal_face_detector() which is used to get the front face from the face image.
modelFile, configFile are the files for the dnn based cv2 face detection model. It is a quantized tensorflow model which is based on the Single Shot-Multibox Detector (SSD) and uses ResNet-10 architecture as its backbone. It was introduced post OpenCV 3.3 in its deep neural network module.
net is the object of the cv2.dnn module that is initialized with the model and config file.
faces=[]name=[]trainpath = "lfw_selected/face"
Here now we are initializing two lists. One for faces and other for names. We will be storing the 128 dimensions face encodings which we get from the face encoder from all the training images into the faces list and the corresding labels in the name list. Here we are also defining the trainpath which will contain the model for the training images.
for im in os.listdir(trainpath):    print(im)    img = cv2.imread(os.path.join(trainpath,im))    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)    frameHeight = img.shape[0]    frameWidth = img.shape[1]    blob = cv2.dnn.blobFromImage(img, 1.0, (300, 300), [104, 117,   123], False, False)    net.setInput(blob)    detections = net.forward()    for i in range(detections.shape[2]):        confidence = detections[0, 0, i, 2]        if confidence > 0.7:        x1 = int(detections[0, 0, i, 3] * frameWidth)        y1 = int(detections[0, 0, i, 4] * frameHeight)        x2 = int(detections[0, 0, i, 5] * frameWidth)        y2 = int(detections[0, 0, i, 6] * frameHeight)        faceAligned =fa.align(img, gray,dlib.rectangle(x1,y1,x2,y2))        landmark = pose_predictor(faceAligned,dlib.rectangle(0,0,faceAligned.shape[0],faceAligned.shape[1]))        face_descriptor = face_encoder.compute_face_descriptor(faceAligned, landmark, num_jitters=2)        faces.append(face_descriptor)        name.append(im)
Here in this code snippet the training of our model will take place. We will iterate over the training folder which contains one image for each person. We will load that image using cv2 and then convert it to gray scale and pass the image to cv2.dnn model to find the face in the image. Then we will iterate over all the detections which are provided by the cv2 dnn model. We have set a confidence threshhold of 0.7 here. Any detection which has confidence value above 0.7 will be taken as the face image and will be taken forward to the Face Aligner for aligning the face. Then we will take out the landmarks from the aligned face and pass the aligned face and landmarks to the face encoder which is the pre trained Resnet model. The pre trained model will give us the (128,1) dimension encoding for all the images and we will keep appending the output with the label to the faces,name list respectively.
faces = np.array(faces)name = np.array(name)np.save('face_repr.npy', faces)np.save('labels.npy', name)
Once we have got encodings for all the images and appended them to list then we will convert that list to numpy array and save that list to the disk using the .npy format. In this way we have saved the list to the disk so that we dont always need to create the list whenever we want to do any face recognition. We can always import the list. We will call the list as model. So at this point we have created and saved the model. Training part has been completed.
faces = np.load("face_repr.npy")name = np.load("labels.npy")image = cv2.imread("lfw_selected/face2/Johnny_Depp_0002.jpg")gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
Now we will start with testing the model which we have created. We will first load our model which we have saved on the disk. Then we will load the image which we want to test the model with and convert it to gray scale.
frameHeight = image.shape[0]frameWidth = image.shape[1]blob = cv2.dnn.blobFromImage(image, 1.0, (300, 300), [104, 117, 123], False, False)net.setInput(blob)detections = net.forward()
Now again we will pass the testing image to the cv2 dnn model and get the detections to detect the face in the image.
scores=[]for i in range(detections.shape[2]):    confidence = detections[0, 0, i, 2]    if confidence > 0.7:        x1 = int(detections[0, 0, i, 3] * frameWidth)        y1 = int(detections[0, 0, i, 4] * frameHeight)        x2 = int(detections[0, 0, i, 5] * frameWidth)        y2 = int(detections[0, 0, i, 6] * frameHeight)        faceAligned = fa.align(image, gray,dlib.rectangle(x1,y1,x2,y2))        landmark = pose_predictor(faceAligned,dlib.rectangle(0,0,faceAligned.shape[0],faceAligned.shape[1]))        face_descriptor = face_encoder.compute_face_descriptor(faceAligned, landmark, num_jitters=2)        score = np.linalg.norm(faces - np.array(face_descriptor), axis=1)        scores.append(score)        imatches = np.argsort(score)        score = score[imatches]        print(name[imatches][:10].tolist(), score[:10].tolist())
Once we have got the face detected in using the cv2 dnn then we will again do the same steps which we performed in the training i.e. forward the image to Face Aligner for aligning the face, take out the landmarks from the aligned face and pass the aligned face and landmarks to the face encoder to generate (128,1) dimension encoding for the image.
After generting the 128 dimension encoding of the image. This 128 dimension vector calculates the euclidean distance of this vector with all the 128-d vectors which are in the faces array which we have calculated before. Then we store all the distance value in the scores array and using the np.argsort we sort the array and take out the 10 smallest values. These smallest value define that these 10 128-d vector have the least euclidian distance from the input image. This implies that the these 10 images are the closest matches to the input image.
Here in the example we had passed the image “Johnny_Depp_0002.jpg” as input and in the output snippet we can see that the least distance image is of “Johnny_Depp_0001.jpg” and the distance is 0.5836. So from this we can conclude that the “Johnny_Depp_0002.jpg” image matches with the “Johnny_Depp_0001.jpg” image. So in this way we done the face recognition with just one image using the One Shot Learning approach.
To make this process even more simple and easy to use we have created a python package that is even more easy to use. We have eliminated all the steps to download the supporting files and setting up the supporting files. You can simply installed the python package and start doing face detection and recognition. To install the package simply using the command written below.
pip install FaceReco
This will download the FaceReco python package into your system.
import FaceReco.FaceReco as frfr_object1 =  fr.FaceReco()fr_object1.train_model("lfw_selected/face")fr_object1.test_model("lfw_selected/face2/Johnny_Depp_0002.jpg")
Here as the package follows the concept of class so we will create object of the class named fr_object1 and call the train_model function and pass the folder containing image to it for training. Once the training is completed then the path where model is saved will be printed to us. Then we can use the test_model function and pass the testing image to the previously trained model for face recogniton. The output will the top 10 matches of the image passed for testing.
We can even load a pretrained model in this package and use it for face recognition.
fr_object2 =  fr.FaceReco()fr_object2.load_model("Model_Object_1")fr_object2.test_model("lfw_selected/face2/Johnny_Depp_0002.jpg")
Here again we have created a new object of the FaceReco class and then using the load_model function we will pass the folder path which contains the saved model to this new class object. In this way the previously saved model is loaded with the object and then we can again do face recognition using the loaded model.
Here by creating this FaceReco python package we have simplified the process of training and testing the face recognition model without any hassles with just single commands. To see the code of this python package you can head over to the link provided below. You can get the fully implemented FaceReco python package from the link provided below.
https://github.com/1297rohit/FaceReco
This repo contains the code for this python package. You can even suggest some new features on this repo.
https://colab.research.google.com/drive/1s7hVaU4WJxyUMDDj1t_Er-uYHGJw73Gr?usp=sharing
This colab link contains the colab notebook with all the code used in this article.