How to create real-time Face Detector | by Dimid | Towards Data Science

In this article, I will show you how to write a real-time face detector using Python, TensorFlow/Keras and OpenCV.
I’ll note right away, that all code files are available in this repo.
gitlab.com
First, in Theoretical Part I will tell you a little about the concepts that will be useful for us (Transfer Learning and Data Augmentation), and then I will go to the code analysis in the Practical Part section.
Note, that you must have TensorFlow and OpenCV libraries installed to run this code. You can do it manually or simply run pip install -r requirements.txt command after you have the repo downloaded. You can also do it with conda commands conda install -c conda-forge numpy, opencv, tensorflow.
Here I want to build a face detector that also can distinguish me from other people. You can see the results of my work at the end of this article.
The initial task can be divided into two subtasks:
Train a model for face classification.Adapt the model to work with a webcam as a detector.
Train a model for face classification.
Adapt the model to work with a webcam as a detector.
In more detail, the task can be divided into the following stages:
Dataset Collection.Search for a Base Model for Transfer Learning.Train a Model.Transform Classifier into Detector.
Dataset Collection.
Search for a Base Model for Transfer Learning.
Train a Model.
Transform Classifier into Detector.
Before I begin to analyze these stages, I would like to very briefly outline the concepts that will be useful for us.
At this point, I must say that this tutorial is not as detailed as it could be. For example, I will not explain what convolution is or why pooling layers are needed in CNN architecture. Of course, you need to know this to understand how CNN works, but I think that this tutorial will be too cumbersome if I explain all the concepts. On the other hand, other people have already explained many things and did it better than me. For example, here you can read about CNN in general, and here — about popular CNN architectures.
Since we are talking about image recognition, we will use CNN — Convolutional Neural Networks, the architecture that has achieved the greatest success in this task.
Unlike classical neural networks (here I mean FDN — Feedforward Neural Network), in CNN, neurons are:
firstly: arranged in the form of a matrix (tensor), not an array,
and secondly: they are connected only with a small number of neurons of the previous layer, so layers are NOT fully connected.
This approach allows you to significantly reduce the number of network parameters and make the search for templates more efficient for images.
At the bottom of this image, I want to explain that the layer in CNN is a tensor (it is often drawn as a parallelepiped). I will talk about a tensor as a sequence of matrices. So each of these tensors is just a series of matrices standing on top of each other like pizza boxes. In turn, matrices are simply a sequence of neurons that are arranged in the form of a matrix, not an array, as in classic networks.
Transfer Learning is the most important concept for this task since it is very difficult (almost impossible) to build a face classifier from scratch. Transfer Learning is the process of using a pretrained model for your own task.
This happens as follows. You find a model that is trained for a similar task (for example, recognizing 1000 classes of images, as in ImageNet, when you need to recognize only a few). Then you use its weights (freeze them, which means that they will not change), and finish training the model on your small dataset. After that, you can also unfreeze all the weights and continue training the model with a very small learning rate.
This process becomes possible due to the features of the CNN architecture and the image recognition task (but also can be used with different architectures). The network learns to see patterns from layer to layer (from input to output) and they are getting more and more complicated. It turns out that the general patterns for all problems are approximately the same, which allows us to use pretrained weights.
Another important concept in the image classification problem is data augmentation. Data augmentation is the process of artificially increasing a dataset size by applying some transformations to the original images. For example, we can use horizontal and vertical reflection, small rotation or magnification, color inversion, and so on. This will significantly increase the size of the dataset and, accordingly, the generalization ability of the network.
towardsdatascience.com
In the example above (first image in the article), when the shift values are quite high, sometimes there is a situation when only part of the image is visible. On the one hand, this is good (the network can learn to recognize a cat not by its muzzle, but by its paws and tail in the case of images where the head is not visible), but it can also confuse the network if you overdo such transformation.
There are several approaches to data augmentation in TensorFlow. We can generate images and save them to disk, or we can feed the generator directly to the network. I chose the first option, to do it more explicitly. But there are two ways here as well: generate a random number of images or generate several versions of a specific image. You can see the implementation of both options in the data_augmentation.ipynb notebook. I used the second option and specifically generated five transformations for each image.
The project has the following structure:
me_not_me_detector├───article├───datasets│   ├───face_dataset_test_images│   │   ├───me      # contains TEST images for ME class│   │   └───not_me  # contains TEST images for NOT_ME class│   ├───face_dataset_train_aug_images (optional)│   │   ├───me      # contains aug TRAIN images for ME class│   │   └───not_me  # contains aug TRAIN images for NOT_ME class│   └───face_dataset_train_images│       ├───me      # contains TRAIN images for ME class│       └───not_me  # contains TRAIN images for NOT_ME class├───models│   .gitignore│   data_augmentation.ipynb│   me_not_me_classifier.ipynb│   me_not_me_classifier_model_comparison.ipynb│   me_not_me_detector.ipynb│   README.md└── requirements.txt
Let’s talk about folders.
The article folder contains the data for this tutorial.
The modelsfolder contains trained models for their test and further use.
The datasets folder contains three folders - for a train set, test set and augmented train set (optional). Each of them contains two subfolders for two classes - me and not_me. In the general case, it contains N subfolders for the N-class classification problem.
Now let’s talk about the code files — jupyter notebooks.
data_augmentation.ipynbfile creates an augmented dataset from an initial one and provides some information about the dataset.
me_not_me_classifier_model_comparison.ipynbfile contains code to train and test five different models.
me_not_me_classifier.ipynbfile does the same thing, but for one particular model. You can use it as an example to build your own classifier.
me_not_me_detector.ipynbfile uses the OpenCV library and turns the classifier into a real-time detector.
Other files are used as additional files:
.gitignore contains files that will not be pushed in git,
requirements.txt contains the list of libraries that have to be installed to run this code, etc.
First of all, we need to collect a dataset. I used my photos and photos of my family to train the model, but you can also train the model on some other face datasets. I used only personal photos to check whether such a small dataset is enough for the model to show acceptable quality.
I cropped the face from each photo and resized them to the size of 250x250 (which you don’t have to do, since TensorFlow can do it for you). Here are some examples of photos from the initial dataset:
In total, I collected 215 photos, where 82 are me and 133 are not_me.
For the test set, I immediately put aside 20 photos. This can be done automatically, but I did it manually because the dataset is too small. For the train set, I applied my data augmentation code from data_augmentation.ipynbnotebook. Thus, the dataset increased five times! (five photos were generated for each photo). Here is an example of the generated photos:
When the dataset is ready, we can start training the model.
There is nothing complicated here — I just used different models that are trained on the ImageNet dataset. If they don’t have good accuracy, then I think I will use something like VGGFace models like here. But the quality I got is good enough. We can load pretrained model with this code:
Now we have to freeze these weights, add some layers on top of the model and train the model on our little dataset.
As I said earlier, me_not_me_classifier_model_comparison.ipynbfile contains the code for different models. You can use it to do your own experiment. Code examples below are taken from me_not_me_classifier.ipynb. We will use ResNet50 as the initial model here.
After import libraries and dataset loading, we load pretrained model exactly as in the code above. After that, we need to restore the top of the model — the dense part. Note, that this part is different for different models. To know what layers do you need, you can load the model with top and without top, call model.summary() method and see, how do they differ:
To add layers I use TensorFlow Functional API. Of course, you need some knowledge about TensorFlow and CNN, but you also can try to add different layers and get non-original architecture. In my case, we need a global average pooling layer and dense layer. Don’t forget to freeze other weights beforehand.
After that, we continue to configure the model. I added a ModelCheckpoint object to save the model on the disk if something goes wrong and also EarlyStopping as a regularization technique. As an optimization method, I used Adam with a standard learning rate value. I also tried using various scheduling techniques, but they did not bring many results.
So now we can start the training process. After it’s over you will have a ready-to-use model. You can check its functionality with the Testing section code in me_not_me_classifier.ipynb file.
Now we have to use OpenCV library for web camera access and face detection.
Frankly speaking, the phrase “transform classifier into detector” is incorrect because of the definitions of the classifier and detector. A classifier (understood as an image classifier) is a model that receives an image as an input, and at the output gives a label of one of the possible classes (in our case, me or not_me). A detector (understood as a face detector) is a model that receives an image as an input and outputs the coordinates of the bounding box around faces if there are faces in this picture.
I use this phrase to show that the final program (which continues to be a classifier) now also gives out exactly where the face is in the picture. In addition, now it can work correctly in the case when there are several faces in the photo at once.
Training a face detector is a very difficult task (mainly due to the fact that the data is difficult to label), so I will use the face detector provided by OpenCV. It is very simple to use. In the code below, the faces variable is an array of all the faces in the image. For each face, there are four values for the bounding box:
x — x coordinate of the upper-left corner,
y — y coordinate of the upper-left corner,
w — width of the bounding box,
h — height of the bounding box.
The following code highlights your face in the webcam image in real-time, as shown in the image below.
Now all that remains is to load the pretrained model, pass it the fragment of the image where the face is located, and display the text on the screen! In fact, at this stage I faced a small difficulty — the model recognized me only sometimes. This happened because I trained her on images “with a margin”, where not only the face was located.
I could add images with a different scale to the dataset, but I decided to do it honestly. To do this, I wrote the function get_extended_image(), which increases the size of the cropped image with a certain coefficient k. The larger the k, the larger the area around the face. To explain the operation of this function (or to confuse you even more), I give the following image (for simplicity, here k = 0.3. The scale is not observed. Note, that the upper left coordinate is (0, 0)). You can also see clippings of my face with different k parameters (according to the OpenCV, it uses a BGR instead of RGB, but instead of adjusting the color, let’s imagine that I am an avatar). For the resulting model, I used k = 0.5.
I want to note that I used MobileNet as a base model. Initially, I used ResNet50, because it showed the best quality, but the image lagged a little, and there are no such problems with the lighter MobileNet. I tested all this on a computer with an i7–10750H and 32G RAM.
That’s it! You can find the finished code in the me_not_me_detector.ipynb notebook, but note that to use it, you need to have a model in the models folder, otherwise you will get an error in the load the model cell. Here is the result.
I must say that although the model is quite simple, it already recognizes me well, for example, if part of the face is covered. Also, I don’t have a photo of Willem Dafoe in the dataset, so the model hasn’t seen him before.
What about face in mask recognition — you can see that if my nose is fully covered, the program stops recognizing my face (or when it is rotated too much). But this is a problem on the detector side because it stops transmitting the image to the model. That is because OpenCV face detectors usually don’t see people in masks. But I suppose that if you can collect a large dataset, you can also teach your model to distinguish people in masks.
What surprised me — it is easier for the model to recognize me when the lower part of the face is covered than the upper part. You can see that when I put on a hat or cover my forehead with my hand, the model begins to doubt. I think this is due to the fact that in all the photos I have my head is open and I have about one hairstyle.
Below I present the results of training different models with or without data augmentation (x5 data augmentation in this case). The best quality (99% validation accuracy) was shown by ResNet50. But as I said above, I used a model based on MobileNet, because it showed itself well in real-time operation.
These results show that data augmentation can significantly increase the quality of the model. In this case validation loss drops 10 times but this is about the very little dataset. I tried to generate more images and got more accuracy. This parameter (the coefficient of the dataset increasing — so the number of augmented images generated for each image) can be chosen with cross-validation. The best model I got among MobileNet’s was one with validation loss = 0.09886 and validation accuracy = 0.9589, and coefficient of the dataset increasing = 15.
At the same time, the learning process became dramatically slower. It makes sense, dataset size increases, and learning time increases too. You can see the plot of the dependence of the training time on the size of the dataset below — it is almost linear.
The next table is the extended version of the previous. Here you can see the number of parameters, depth of the model, and test accuracy. Note that layers are counted as the number of model.layers objects, which is not quite correct, but gives an idea of the depth of the models. Test accuracy doesn’t have a lot of statistical power, because I had only 20 test images.
One thing about VGG. I replaced two 4096 dense layers with two 1024 dense layers, and the model showed approximately the same result, although the number of parameters fell in three times (from 134 million to 41 million, and the trainable — from 119 million to 27 million). The size on the disk is 368 MB for this model.
From the work done, I can make the following conclusions. Some of them are more obvious, some of them are less:
The dataset that you need to collect should be as diverse as possible and at the same time as similar as possible to real data.
More simple models are training faster, making predictions faster and taking less place on the hard disk. More simple models have fewer parameters, but, at the same time, can be deeper.
Transfer Learning is a very powerful technique that allows you to solve your task even with a tiny dataset.
Data Augmentation technique allows you to significantly increase the quality of your models, but it also increases training time. You can choose a data augmentation coefficient with cross-validation.
The result strongly depends on random initialization. Therefore, even the training process with the same parameters should be run several times. For example, in 22 minutes and 150 epochs, I got the result 0.1334 for validation loss with ResNet50 without data augmentation, which is 39% better than regular ResNet50.
Now I will say one more phrase that will confuse you even more.
The convolutional part of the network (up to dense layers) transforms the original image into a point in some high-dimensional space (this is similar to the concept of embedding), and dense layers then build a separating hyperplane in this space.
So, it is better to build a model that will better locate these points, and then you can do with a more simple classifier. Of course, the rules by which these representations are built are very complex, which is why we need deep models with a large number of parameters.
A network that creates embeddings well can solve tasks for which it was not originally trained. So, a good model will locate all my photos close to each other (in some high-dimensional space) and all Willem Dafoe’s photos close to each other. But at the same time, these two clouds of points should be located far apart. Now, when you show a network a photo of another person, it will be able to say that it is not_me, but in addition, it will be able to give out the person he most resembles or note that this person was not in the dataset. This is a variation of Metric Learning, Triplet Loss is used here.
I hope these materials were useful to you. Follow me on Medium to get more articles like this.
If you have any questions or comments, I will be glad to get any feedback. Ask me in the comments, or connect via LinkedIn or Twitter.
To support me as a writer and to get access to thousands of other Medium articles, get Medium membership using my referral link (no extra charge for you).