Title: What is face recognition? - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/01/what-is-face-recognition/

Published Time: 2021-05-01T14:00:00+00:00

Markdown Content:
In this tutorial, you will learn about face recognition, including:

*   How face recognition works
*   How face recognition is _different_ from face detection
*   A history of face recognition algorithms
*   State-of-the-art algorithms used for face recognition today

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20467'%3E%3C/svg%3E)

Next week we will start implementing these face recognition algorithms.

**To learn about face recognition, _just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

Face recognition is the process of taking a face in an image and actually _identifying_ who the face belongs to. Face recognition is thus a form of **person identification.**

Early face recognition systems relied on an early version of [facial landmarks](https://pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/) extracted from images, such as the relative position and size of the eyes, nose, cheekbone, and jaw. However, these systems were often highly subjective and prone to error since these quantifications of the face were manually extracted by the computer scientists and administrators running the face recognition software.

As machine learning algorithms became more powerful and the computer vision field matured, face recognition systems started to utilize feature extraction and classification models to identify faces in images.

Not only are these systems non-subjective, but they are also automatic — no hand labeling of the face is required. We simply extract features from the faces, train our classifier, and then use it to identify subsequent faces.

Most recently, we’ve started to [utilize deep learning algorithms for face recognition](https://pyimagesearch.com/2018/09/24/opencv-face-recognition/). State-of-the-art face recognition models such as FaceNet and OpenFace rely on a specialized deep neural network architecture called **siamese networks**.

These neural networks are capable of obtaining face recognition accuracy that was once thought impossible (and they can achieve this accuracy with surprisingly little data).

### **How is face** **_recognition_** **different from face** **_detection?_**

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20441'%3E%3C/svg%3E)

**Figure 1:** Face recognition can be thought of as a two-step process. First, we must detect the presence of the face using a face detector and extract the face region of interest (ROI). Once we have the face ROI we can perform face recognition, the process of actually _identifying_ who is in the image.

I’ve often seen new computer vision and deep learning practitioners confuse the difference between face detection and face recognition, sometimes (and incorrectly) using the terms interchangeably.

**Face detection and face recognition are** **_distinctly different_** **algorithms** — **_face detection_** **will tell you** **_where_** **in a given image/frame a face is (but not** **_who_** **the face belongs to) while** **_face recognition_** **actually** **_identifies_** **the detected face.**

Let’s break this down a bit farther:

Unlike [**face detection**](https://pyimagesearch.com/2018/02/26/face-detection-with-opencv-and-deep-learning/), which is the process of simply _detecting the presence_ of a face in an image or video stream, **face recognition** takes the faces detected from the localization phase and attempts to identify _whom_ the face belongs to. Face recognition can thus be thought of as a method of person identification, which we use heavily in security and surveillance systems.

**Since face recognition, by definition, requires face detection, we can think of face recognition as a two-phase process.**

*   **Phase #1:** Detect the presence of faces in an image or video stream using methods such as Haar cascades, HOG + Linear SVM, deep learning, or any other algorithm that can localize faces.
*   **Phase #2:** Take each of the faces detected during the localization phase and identify each of them — this is where we actually assign a name to a face.

In the remainder of this tutorial, we’ll review a quick history of face recognition, followed by introducing face recognition algorithms and techniques, including Eigenfaces, Local Binary Patterns (LBPs) for face recognition, siamese networks, FaceNet, etc.

### **A brief history of face recognition**

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20353'%3E%3C/svg%3E)

**Figure 2:** Face recognition was first conceived in the early 1970s but it wasn’t until 2015 that we achieved robust face recognition accuracy, much in part due to deep learning and siamese network architectures ([image source](https://www.researchgate.net/figure/Major-milestones-in-the-history-of-automated-face-recognition-While-the-events-in-the_fig9_290509735)).

Face recognition may seem ubiquitous now (with it being implemented on most smartphones and major social media platforms), but prior to the 1970s, face recognition was often regarded as science fiction, sequestered to the movies and books set in ultra-future times. In short, face recognition was a fantasy, and whether or not it would become a reality was unclear.

This all changed in 1971 when Goldstein et al. published [_Identification of human faces_](https://ieeexplore.ieee.org/document/1450184). A crude first attempt at face identification, this method proposed 21 subjective facial features, such as hair color and lip thickness, to identify a face in a photograph.

The largest drawback of this approach was that the 21 measurements (besides being highly subjective) were _manually computed_ — an obvious flaw in a computer science community that was rapidly approaching unsupervised computation and classification (at least in terms of human oversight).

Then, over a decade later, in 1987, Sirovich and Kirby published their seminal work, [_A Low-Dimensional Procedure for the Characterization of Human Faces_](https://pyimagesearch.com/wp-content/uploads/2021/05/kirby_1987.pdf) which was later followed by Turk and Pentland in 1991 with [_Face Recognition Using Eigenfaces_](https://sites.cs.ucsb.edu/~mturk/Papers/mturk-CVPR91.pdf)_._

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20523%20500'%3E%3C/svg%3E)

**Figure 3:** The Eigenfaces algorithm is rooted in linear algebra and Principal Component Analysis (PCA) ([image source](https://towardsdatascience.com/eigenfaces-recovering-humans-from-ghosts-17606c328184)).

Both Sirovich and Kirby, along with Turk and Pentland, demonstrated that a standard linear algebra technique for dimensionality reduction called [Principal Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) (PCA) could be used to identify a face using a feature vector smaller than 100-dim.

Furthermore, the “principal components” (i.e., the eigenvectors, or the “eigenfaces”) could be used to reconstruct faces from the original dataset. This implies that a face could be represented (and eventually identified) as a linear combination of the eigenfaces:

_Query Face = 36% of Eigenface #1 + -8% of Eigenface #2 … + 21% of Eigenface N_

Following the work of Sirovich and Kirby in the late 1980s, further research in face recognition exploded — another popular linear algebra-based face recognition technique utilized [Linear Discriminant Analysis](https://en.wikipedia.org/wiki/Linear_discriminant_analysis). LDA-based face recognition algorithms are commonly known as **Fisherfaces**.

Feature-based approaches such as [**Local Binary Patterns**](https://pyimagesearch.com/2015/12/07/local-binary-patterns-with-python-opencv/) **for face recognition** have also been proposed and are still heavily used in real-world applications:

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20370'%3E%3C/svg%3E)

**Figure 4:** Local Binary Patterns (LBPs) can be used for face recognition ([image source](https://www.researchgate.net/figure/LBP-face-recognition_fig2_258936085)).

Deep learning is now responsible for _unprecedented_ accuracy in face recognition. Specialized architectures called **siamese networks** are trained with a special type of data, called **image triplets**. We then compute, monitor, and attempt to minimize our **triplet loss**, thereby maximizing face recognition accuracy.

Popular deep neural network face recognition models include [FaceNet](https://arxiv.org/abs/1503.03832) and [OpenFace](http://cmusatyalab.github.io/openface/).

### **Eigenfaces**

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20326'%3E%3C/svg%3E)

**Figure 5:** The Eigenfaces algorithm used for face recognition ([image source](https://www.geeksforgeeks.org/ml-face-recognition-using-eigenfaces-pca-algorithm/)).

The Eigenfaces algorithm uses Principal Component Analysis to construct a low-dimensional representation of face images.

This process involves collecting a dataset of faces with multiple face images per person we want to identify (like having multiple training examples of an image class we want to identify when performing image classification).

Given this dataset of face images, presumed to be the same width, height, and ideally — with their eyes and facial structures aligned at the same _(x, y)_\-coordinates, we apply an eigenvalue decomposition of the dataset, keeping the eigenvectors with the largest corresponding eigenvalues.

Given these eigenvectors, a face can then be represented as a linear combination of what Sirovich and Kirby call **eigenfaces.**

Face identification can be performed by computing the Euclidean distance between the eigenface representations and treating the face identification as a k-Nearest Neighbor classification problem — however, we tend to commonly apply more advanced machine learning algorithms to the eigenface representations.

If you’re feeling a bit overwhelmed by the linear algebra terminology or how the Eigenfaces algorithm works, no worries — we’ll be covering the Eigenfaces algorithm in detail later in this series of tutorials on face recognition.

### **LBPs for face recognition**

While the Eigenfaces algorithm relies on PCA to construct a low-dimensional representation of face images, the Local Binary Patterns (LBPs) method relies, as the name suggests, on feature extraction.

First introduced by Ahonen et al. in their 2004 paper, [_Face Recognition with Local Binary Patterns_](https://link.springer.com/chapter/10.1007/978-3-540-24670-1_36), their method suggests dividing a face image into a _7×7_ grid of equally sized cells:

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20452%20500'%3E%3C/svg%3E)

**Figure 6:** Once a face has been detected in an image, the first step is to divide the face ROI into _7×7_ equally sized cells. We then extract a [Local Binary Pattern](https://pyimagesearch.com/2015/12/07/local-binary-patterns-with-python-opencv/) histogram from each of the 49 cells.

By dividing the image into cells we can introduce _locality_ into our final feature vector. Furthermore, some cells are weighted such that they contribute _more_ to the overall representation. **Cells in the corners carry less identifying facial information compared to the cells in the center of the grid (which contain eyes, nose, and lip structures).**

Finally, we concatenate the weighted LBP histograms from the 49 cells to form our final feature vector.

The actual face identification is performed by k-NN classification using the ![Image 9: \chi^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2016%2017'%3E%3C/svg%3E) distance between the query image and the dataset of labeled faces — since we are comparing histograms, the ![Image 10: \chi^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2016%2017'%3E%3C/svg%3E) distance is a better choice than the Euclidean distance.

While both Eigenfaces and LBPs for face recognition are fairly straightforward algorithms for face identification, the feature-based LBP method tends to be more resilient against noise (since it does not operate on the raw pixel intensities themselves) and will usually yield better results.

We’ll be implementing LBPs for face recognition in detail later in this series of tutorials.

### **Deep learning-based face recognition**

![Image 11](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20496'%3E%3C/svg%3E)

**Figure 7:** Deep learning face recognition is responsible for unprecedented accuracy in the computer vision and face recognition literature ([image source](https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78)).

Deep learning has impacted nearly every aspect and subfield of computer science. Face recognition is no different.

For many years, LBPs and Eigenfaces/Fisherfaces were considered the state-of-the-art in face recognition. These techniques were easily fooled, and accuracy was poor outside the research lab/controlled environments.

Deep learning changed all that. Specialized neural network architectures and training techniques, including **siamese networks**, **image triplets**, and **triplet loss,** enabled researchers to obtain face recognition accuracy that was once thought impossible.

These methods are _far_ more accurate and robust than previous techniques. And despite the stigma of neural networks being data hungry beasts, siamese networks allow us to train these state-of-the-art models with very little data.

If you’re interested in learning more about deep learning-based face recognition, I suggest you read the following guides on PyImageSearch:

1.  [_Face recognition with OpenCV, Python, and deep learning_](https://pyimagesearch.com/2018/06/18/face-recognition-with-opencv-python-and-deep-learning/)
2.  [_OpenCV Face Recognition_](https://pyimagesearch.com/2018/09/24/opencv-face-recognition/)
3.  [_Raspberry Pi Face Recognition_](https://pyimagesearch.com/2018/06/25/raspberry-pi-face-recognition/)
4.  [_Raspberry Pi and Movidius NCS Face Recognition_](https://pyimagesearch.com/2020/01/06/raspberry-pi-and-movidius-ncs-face-recognition/)

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial, we learned that face recognition is a two-phase process that consists of:

1.  Face detection and extraction of the face ROI (region of interest)
2.  Identification, where we identify _who_ the face belongs to

From there, we reviewed the history of face recognition algorithms, including:

*   Crude (and often subjective) facial landmarks that were manually labeled by researchers
*   Linear algebra-based techniques such as Eigenfaces and Fisherfaces
*   LBPs for face recognition
*   Deep learning-based models, including FaceNet and OpenFace

The next tutorial in this series will cover how to implement Eigenfaces with OpenCV.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch),** **_simply enter your email address in the form below!_**

![Image 12](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.