Building a Face Recognition System Using Scikit Learn in Python | by Dr. Varshita Sher | Towards Data Science

Face recognition is the task of comparing an unknown individual’s face to images in a database of stored records. The mapping could be one–to–one or one–to–many, depending on whether we are running face verification or face identification.
In this tutorial, we are interested in building a facial identification system that will verify if an image, generally known as probe image, exists within a pre-existing database of faces, generally known as the evaluation set.
There are four main steps involved in building such a system:
Available face detection models include MTCNN, FaceNet, Dlib, etc.
OpenCV library provides all the tools we need for this step.
Since programs can’t work with jpg or png files directly, we need some way of translating images to numbers. In this tutorial, we will be using the Insightface model for creating a multi-dimensional (512-d) embedding for a face such that it encapsulates useful semantic information pertaining to the face.
To tackle all three steps using a single library, we will be using insightface. In particular, we will be working with Insightface’s ArcFace model.
InsightFace is an open-sourced deep face analysis model for face recognition, face detection and face align-ment tasks.
Once we have translated each unique face into a vector, comparing faces essentials boils down to comparing the corresponding embeddings. We will be making use of these embeddings to train a sci-kit learn model.
P.S. If you’d like to follow along, the code is available on Github.
Create a virtual environment (optional):
python3 -m venv face_search_env
Activate this environment:
source face_search_env/bin/activate
Necessary installations within this environment:
pip install mxnet==1.8.0.post0pip install -U insightface==0.2.1pip install onnx==1.10.1pip install onnxruntime==1.8.1
More importantly, once you are done with pip installing insightface:
- Download the antelope model release from onedrive. (It contains two pre-trained models for detection and recognition).- Put it under ~/.insightface/models/, so there're onnx models at ~/.insightface/models/antelope/*.onnx.
This is how it should look like if the setup was done correctly:
and if you look inside the antelope directory, you’ll find the two onnx models for face detection and recognition:
Note: Since the latest release of insightface 0.4.1 last week, the installation was not as straightforward as I would have hoped (at least for me). Hence, I will be using 0.2.1 for this tutorial. In the future, I’ll update the code on Github accordingly. Please see the instructions here if you’re stuck.
We will be working with the Yale Faces dataset available on Kaggle, containing approximately 165 grayscale images of 15 individuals (i.e. 11 unique images per identity). The images are composed of a wide variety of expressions, poses, and illumination configurations.
Once you have the dataset, go ahead and unzip it inside a newly createddata directory within your project (see the project directory structure on Github).
If you’d like to follow along, the Jupyter Notebook can be found on Github.
import osimport pickleimport numpy as npfrom PIL import Imagefrom typing import Listfrom tqdm import tqdmfrom insightface.app import FaceAnalysisfrom sklearn.neighbors import NearestNeighbors
Once insightface is installed, we must call app=FaceAnalysis(name="model_name")to load the models.
Since we stored our onnx models inside the antelope directory:
app = FaceAnalysis(name="antelope")app.prepare(ctx_id=0, det_size=(640, 640))
Generating an embedding for an image is quite straightforward with the insightface model. For instance:
# Generating embeddings for an imageimg_emb_results = app.get(np.asarray(img))img_emb = img_emb_results[0].embeddingimg_emb.shape------------OUTPUT---------------(512,)
Prior to using this dataset, we must fix the extensions for the files in the directory such that file names end with .gif. (or .jpg , .png, etc).
For instance, the following code snippet will change the filename subject01.glasses to subject01_glasses.gif.
Next, we will split the data into the evaluation and probe sets: 90% or 10 images per subject will become part of the evaluation set and the remaining 10% or 1 image per subject will be used in the probe set.
To avoid sampling bias, the probe image for each subject will be randomly chosen using a helper function called create_probe_eval_set() . It takes as input a list containing the (file names for the) 11 images belonging to a particular subject and returns two lists of lengths 1 and 10. The former contains the filename to be used for the probe set while the latter contains file names for the evaluation set.
Both the lists returned by the create_probe_eval_set() are sequentially fed to a helper function called generate_embs(). For each filename in the list, it reads the grayscale image, converts it to RGB, calculates the corresponding embeddings, and finally returns the embeddings along with the image labels (scraped from the filename).
Now that we have a framework for generating embeddings, let’s go ahead and create embeddings for both probe and evaluation set using generate_embs().
Few things to consider:
The files returned by os.listdir()are in completely random order, hence sorting on line 3 is important. Why do we need sorted filenames? Remember create_probe_eval_set() on line 11 requires all files belonging to a particular subject in any single iteration.
[Optional] We could have replaced the create_probe_eval_set() function, get rid of the forloop, and simplified a few lines in the above code snippet if we used the stratified train_test_splitfunctionality provided by sklearn. For this tutorial, however, I prioritized clarity over code simplicity.
Oftentimes, insightface is unable to detect a face and subsequently generates an empty embedding for it. That explains why some of the entries in probe_setor eval_set list might be empty. It is important that we filter them out and keep only non-empty values.
To do so, we create another helper function called filter_empty_embs():
It takes as input the image set (either probe_set or eval_set ) and removes those elements for which insightface could not generate an embedding (see Line 6). Following this, it also updates the labels (either probe_labelsor eval_labels) (see Line 7) such that both sets and labels have the same length.
Finally, we can obtain the 512-d embeddings for only the good indices in both evaluation set and probe set:
assert len(evaluation_embs) == len(evaluation_labels)assert len(probe_embs) == len(probe_labels)
With both sets at our disposal, we are now ready to build our face identification system using a popular unsupervised learning method implemented in the Sklearn library.
We train the Nearest neighbor model using .fit() with evaluation embeddings as X. This is a neat technique for unsupervised nearest neighbors learning.
The nearest neighbour method allows us to find a predefined number of training samples closest in distance to a new point.
Note: The distance can, in general, be any metric measure such as Euclidean, Manhattan, Cosine, Minkowski, etc.
Because we are implementing an unsupervised learning method, observe that we do not pass any labels, i.e. evaluation_label to the fit method. All we are doing here is mapping out the face embeddings in the evaluation set into a latent space.
Why??, you ask. Simple answer: By storing the training set in memory ahead of time, we are able to speed up the search for its nearest neighbors during inference time.
How does it do this? Simple answer: Storing the tree in an optimized manner in memory is quite useful, especially when the training set is large and searching for a new point’s neighbors becomes computationally expensive.
Neighbors-based methods are known as non-generalizing machine learning methods, since they simply “remember” all of its training data (possibly transformed into a fast indexing structure such as a Ball Tree or KD Tree). [Source]
Note: See this Stackoverflow discussion if you are still not convinced!
For each new probe image, we can find whether it is present in the evaluation set by searching for its top k neighbors using nn.neighbours()method. For instance,
# Example inference on test imagedists, inds = nn.kneighbors(X = probe_img_emb.reshape(1,-1),                            n_neighbors = 3,                            return_distances = True                            )
If the labels at the returned indices (inds) in the evaluation set are a perfect match for the probe image’s original/true label, then we know we have found our face in the verification system.
We have wrapped the aforementioned logic into the print_ID_results() method. It takes as input the probe image path, the evaluation set labels, and the verbose flag to specify if detailed results should be displayed.
Few important things to note here:
inds contain the indices of the nearest neighbors in the evaluation_labels set (line 6). For instance, inds = [[2,0,11]]means label at index=2 in evaluation_labels is found to be nearest to the probe image, followed by the label at index = 0.
Since for any image, nn.neighborswill return a non-empty response, we must only consider those results as a face ID match if the distance returned is less than or equal to 0.6 (line 12). (P.S. The choice of 0.6 is completely arbitrary).For example, continuing with the above example where inds = [[2,0, 11]]and let's saydists = [[0.4, 0.6, 0.9]], we will only consider the labels at index=2 and index = 0 (in evaluation_labels) as a true face match because the dist for the last neighbor is too large for it to be a genuine match.
As a quick sanity check, let’s see the system’s response when we input a baby’s face as a probe image. As expected, it reveals no matching faces found! However, we set verbose as True, because of which we get to see the labels and distances for its bogus nearest neighbors in the database, all of which appear to be quite large (>0.8).
One of the ways to test whether this system is any good is to see how many relevant results are present in the top k neighbors. A relevant result is one where the true label matches the predicted label. This metric is generally referred to as precision at k, where k is predetermined.
For instance, pick an image (or rather an embedding ) from the probe set with a true label as ‘subject01’. If the top two pred_labels returned by nn.neighborsfor this image are [‘subject01’, ‘subject01’], it means the precision at k (p@k) with k=2 is 100%. Similarly, if only one of the values in pred_labels was equal to ‘subject05’, p@k would be 50%, and so on...
dists, inds = nn.kneighbors(X=probe_embs_example.reshape(1, -1),                            n_neighbors=2,                            return_distance=True)pred_labels = [evaluation_labels[i] for i in inds[0] ]pred_labels----- OUTPUT ------['002', '002']
Let’s go ahead and calculate the average p@k value across the entire probe set:
Awesome! 90% Not too shabby but definitely could be improved (but that’s for another time)...
Kudos to you for following this through! Hopefully, this warm introduction to face recognition, an active area of research in computer vision, was enough to get you started. As always, if there’s an easier way to do some of the things I mentioned in this article, please do let me know.
Until next time :)