Title: OpenCV Eigenfaces for Face Recognition - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/10/opencv-eigenfaces-for-face-recognition/

Published Time: 2021-05-10T14:00:00+00:00

Markdown Content:
In this tutorial, you will learn how to implement face recognition using the Eigenfaces algorithm, OpenCV, and scikit-learn.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20450'%3E%3C/svg%3E)

Our [previous tutorial](https://pyimagesearch.com/2021/05/03/face-recognition-with-local-binary-patterns-lbps-and-opencv/) introduced the concept of face recognition — detecting the presence of a face in an image/video and then subsequently identifying the face.

We’re now going to learn how to utilize linear algebra, and more specifically, principal component analysis, to recognize faces.

This algorithm is important to understand from both a theoretical and historical perspective, so make sure you fully read the guide and digest it.

**To learn how to implement Eigenfaces for face recognition,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this tutorial, we’ll discuss the Eigenfaces algorithm, including how it utilizes linear algebra and Principal Component Analysis (PCA) to perform face recognition.

From there we’ll configure our development environment and then review our project directory structure.

I’ll then show you how to implement Eigenfaces for face recognition using OpenCV and scikit-learn.

Let’s get started!

### **Eigenfaces, Principal Component Analysis (PCA), and face recognition**

Fundamentals of the Eigenfaces algorithm were first presented by Sirovich and Kirby in their 1987 paper, _[Low-Dimensional Procedure for the Characterization of Human Faces](https://pyimagesearch.com/wp-content/uploads/2021/05/kirby_1987.pdf),_ and then later formalized by Turk and Pentland in their 1991 CVPR paper, [_Face Recognition Using Eigenfaces_](https://sites.cs.ucsb.edu/~mturk/Papers/mturk-CVPR91.pdf)_._

These papers are considered to be a seminal work in the history of computer vision — and while other approaches have since been proposed that can outperform Eigenfaces, it’s still important that we take the time to understand and appreciate this algorithm. We’ll be discussing the technical inner workings of Eigenfaces here today.

The first step in the Eigenfaces algorithm is to input a dataset of _N_ face images:

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20260'%3E%3C/svg%3E)

**Figure 1:** A sample of our CALTECH Faces dataset.

For face recognition to be successful (and somewhat robust), we should ensure we have multiple images per person we want to recognize.

Let’s now consider an image containing a face:

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20416%20450'%3E%3C/svg%3E)

**Figure 2:** Each face in our image is considered to be a _KxK_ image (although non-square face images will certainly work as well, we’ll just use square images to make this lesson a bit easier to explain).

When applying Eigenfaces, each face is represented as a grayscale, _K×K_ bitmap of pixels (images do not have to be square, but for the sake of this example, it’s easier to explain if we assume square images).

In order to apply the Eigenfaces algorithm, we need to form a _single vector_ from the image. This is accomplished by “flattening” each image into a ![Image 5: K^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2021%2014'%3E%3C/svg%3E)\-dim vector:

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20241'%3E%3C/svg%3E)

**Figure 3:** We can take our _K×K_ image and then flatten it into a single “feature vector” of raw pixel intensities.

Again, all we have done here is taken a _K×K_ image and concatenated all of the rows together, forming a single, long ![Image 7: K^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2021%2014'%3E%3C/svg%3E) list of grayscale pixel intensities.

After each image in the dataset has been flattened, we form a matrix of flattened images like this, where _Z_ is the total number of images in our dataset:

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20449%20400'%3E%3C/svg%3E)

**Figure 4:** Our entire CALTECH Faces dataset represented as a matrix with _Z_ rows, the number of images, and _K_\-squared columns, the flattened image representation.

**Our entire dataset is now contained in a single matrix,** **_M._**

Given this matrix _M_, we are now ready to apply [Principal Component Analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis), the cornerstone of the Eigenfaces algorithm.

A complete review associated with the linear algebra underlying PCA is outside the scope of this lesson (for a detailed review of the algorithm, please see [Andrew Ng’s discussion on the topic](https://www.coursera.org/learn/machine-learning)), but the general outline of the algorithm follows:

1.  Compute the mean ![Image 9: \mu_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2014%2010'%3E%3C/svg%3E) of each column in the matrix, giving us the average pixel intensity value for **_every_** _(x, y)_\-coordinate in the image dataset.
2.  Subtract the ![Image 10: \mu_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2014%2010'%3E%3C/svg%3E) from each column ![Image 11: c_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2011%2010'%3E%3C/svg%3E) — this is called _mean centering_ the data and is a required step when performing PCA.
3.  Now that our matrix _M_ has been mean centered, compute the _covariance matrix._
4.  Perform an eigenvalue decomposition on the covariance matrix to get the eigenvalues ![Image 12: \lambda_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2014'%3E%3C/svg%3E) and eigenvectors ![Image 13: \mathbf{X_{i}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2018%2016'%3E%3C/svg%3E).
5.  Sort ![Image 14: \mathbf{X_{i}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2018%2016'%3E%3C/svg%3E) by ![Image 15: |\lambda_{i}|](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2021%2017'%3E%3C/svg%3E), largest to smallest.
6.  Take the top _N_ eigenvectors with the largest corresponding eigenvalue magnitude.
7.  Transform the input data by projecting (i.e., taking the dot product) it onto the space created by the top _N_ eigenvectors — **these eigenvectors are called our** **_eigenfaces._**

Again, a complete review on manually computing the covariance matrix and performing an eigenvalue decomposition is outside the scope of this lesson. For a more detailed review, please see [Andrew Ng’s machine learning lessons](https://www.coursera.org/learn/machine-learning) or consult this excellent [Principal Components Analysis primer](http://www.iro.umontreal.ca/~pift6080/H09/documents/papers/pca_tutorial.pdf) by Lindsay Smith.

However, before we perform actual face identification using the Eigenfaces algorithm, let’s actually discuss these eigenface representations:

Each row in the matrix above is an eigenface with ![Image 16: K^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2021%2014'%3E%3C/svg%3E) entries — _exactly like our original image_

What does this mean? Well, since each of these eigenface representations is actually a ![Image 17: K^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2021%2014'%3E%3C/svg%3E) vector, we can reshape it into a _K×K_ bitmap:

![Image 18](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20466'%3E%3C/svg%3E)

**Figure 6:** _Left:_ Mean face image. _Right:_ The top 16-eigenface representations. Light regions demonstrate high variation in the dataset whereas dark areas suggest little variation.

The image on the _left_ is simply the average of all faces in our dataset, while the figures on the _right_ show the most prominent deviations from the mean in our face dataset.

**This can be thought of as a visualization of the dimension in which people’s faces vary the most.** Lighter regions correspond to higher variation, where darker regions correspond to little to no variation. Here, we can see that our eigenface representation captures considerable variance in the eyes, hair, nose, lips, and cheek structure.

Now that we understand how the Eigenfaces representation is constructed, let’s move on to learning how we can actually _identify_ faces using Eigenfaces.

### **Identifying faces using Eigenfaces**

Given our eigenface vectors, we can represent a new face by taking the dot product between the (flattened) input face image and the _N_ eigenfaces. This allows us to represent each face as a linear combination of principal components:

_Query Face = 36% of Eigenface #1 + -8% of Eigenface #2 + … + 21% of Eigenface N_

To perform the actual face identification, Sirovich and Kirby proposed taking the Euclidean distance between projected eigenface representations — this is, in essence, a k-NN classifier:

![Image 19](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20481%20475'%3E%3C/svg%3E)

**Figure 7:** Computing the Euclidean distance (denoted as function, _d_) between face representations. **The smaller the distance, the more similar the faces are considered to be.**

The smaller the Euclidean distance (denoted as the function, _d_), the more “similar” the two faces are — the overall identification is found by taking the label associated with the face with the smallest Euclidean distance.

For example, in **Figure 7** the top image pair has a distance of `0` because the two faces are _identical_ (i.e., the same image).

The middle image pair has a distance of `0.07` — while the images are different they contain the same face.

The third image pair has a much larger distance (`9.81`), indicating that the two faces presented to the Eigenfaces algorithm are _not_ the same person.

In practice, we often don’t rely on a simple k-NN algorithm for identification. Accuracy can be increased by using more advanced machine learning algorithms, such as Support Vector Machines (SVMs), Random Forests, etc. The implementation covered here today will utilize SVMs.

### **Configuring your development environment**

To learn how to use the Eigenfaces algorithm for face recognition, you need to have OpenCV, scikit-image, and scikit-learn installed on your machine:

Luckily, OpenCV is pip-installable:

$ pip install opencv-contrib-python
$ pip install scikit-image
$ pip install scikit-learn

**If you need help configuring your development environment for OpenCV, I _highly recommend_ that you read my** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in a matter of minutes.

### **Having problems configuring your development environment?**

[![Image 20](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 8**: Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux systems?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

### **The CALTECH Faces dataset**

![Image 21](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20480'%3E%3C/svg%3E)

**Figure 9:** A sample of the CALTECH Faces dataset.

The CALTECH Faces challenge is a benchmark dataset for face recognition algorithms. Overall, the dataset consists of 450 images of approximately 27 unique people. Each subject was captured under various lighting conditions, background scenes, and facial expressions, as seen in **Figure 9**.

The overall goal of this tutorial is to apply the Eigenfaces face recognition algorithm to _identify each of the subjects_ in the CALTECH Faces dataset.

**_Note: I’ve included a slightly modified version of the CALTECH Faces dataset in the “Downloads” associated with this tutorial._** _The slightly modified version includes an easier to parse directory structure with faux names assigned to each of the subjects, making it easier to evaluate the accuracy of our face recognition system. Again, you_ **_do not_** _need to download the CALTECH Faces dataset from CALTECH’s servers — just use the “Downloads” associated with this guide._

### **Project structure**

Before we can implement Eigenfaces with OpenCV, let’s first review our project directory structure.

Be sure to access the **_“Downloads”_** section of this tutorial to retrieve the source code, pre-trained face detector model, and CALTECH faces dataset.

After unarchiving the .zip you should have the following structure:

$ tree --dirsfirst --filelimit 20
.
├── caltech\_faces \[26 entries exceeds filelimit, not opening dir\]
├── face\_detector
│   ├── deploy.prototxt
│   └── res10\_300x300\_ssd\_iter\_140000.caffemodel
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   └── faces.py
└── eigenfaces.py

4 directories, 7 files

Our project directory structure is essentially identical to that of last week when we discussed implementing [face recognition with Local Binary Patterns (LBPs)](https://pyimagesearch.com/2021/05/03/face-recognition-with-local-binary-patterns-lbps-and-opencv/):

*   `face_detector`: Stores [OpenCV’s pre-trained deep learning-based face detector](https://pyimagesearch.com/2018/02/26/face-detection-with-opencv-and-deep-learning/)
*   `pyimagesearch`: Contains the `detect_faces` and `load_face_dataset` helper functions which perform face detection and load our CALTECH Faces dataset from disk, respectively
*   `eigenfaces.py`: Our driver script that trains the Eigenfaces model

The `caltech_faces` directory is structured as follows:

$ ls -l caltech\_faces/
abraham
alberta
allen
carmen
conrad
cynthia
darrell
flyod
frank
glen
gloria
jacques
judy
julie
kathleen
kenneth
lewis
mae
phil
raymond
rick
ronald
sherry
tiffany
willie
winston

$ ls -l caltech\_faces/abraham/\*.jpg
caltech\_faces/abraham/image\_0022.jpg
caltech\_faces/abraham/image\_0023.jpg
caltech\_faces/abraham/image\_0024.jpg
...
caltech\_faces/abraham/image\_0041.jpg

Inside this directory is a subdirectory containing images for each of the people we want to recognize. As you can see, we have multiple images for each person we want to recognize. These images will serve as our training data such that our LBP face recognizer can learn what each individual looks like.

### **Implementing face detection and CALTECH face dataset loading**

In this section, we’ll be implementing two functions that will facilitate working with the CALTECH Faces dataset:

1.  `detect_faces`: Accepts an input image and performs face detection, returning the bounding box _(x, y)_\-coordinates of all faces in the image
2.  `load_face_dataset`: Loops over all images in the CALTECH Faces dataset, performs face detection, and returns both the face ROIs and class labels (i.e., names of the individuals) to the calling function

Both of these functions were covered in detail in last week’s tutorial on [_Face Recognition with Local Binary Patterns (LBPs) and OpenCV_](https://pyimagesearch.com/2021/05/03/face-recognition-with-local-binary-patterns-lbps-and-opencv/). I’ve included both of these functions here today as a matter of completeness, but you should refer to the previous article for more details on them.

With that said, open `faces.py` inside the `pyimagesearch` module and let’s take a peek at what’s going on:

\# import the necessary packages
from imutils import paths
import numpy as np
import cv2
import os

def detect\_faces(net, image, minConfidence=0.5):
	# grab the dimensions of the image and then construct a blob
	# from it
	(h, w) = image.shape\[:2\]
	blob = cv2.dnn.blobFromImage(image, 1.0, (300, 300),
		(104.0, 177.0, 123.0))

	# pass the blob through the network to obtain the face detections,
	# then initialize a list to store the predicted bounding boxes
	net.setInput(blob)
	detections = net.forward()
	boxes = \[\]

	# loop over the detections
	for i in range(0, detections.shape\[2\]):
		# extract the confidence (i.e., probability) associated with
		# the detection
		confidence = detections\[0, 0, i, 2\]

		# filter out weak detections by ensuring the confidence is
		# greater than the minimum confidence
		if confidence > minConfidence:
			# compute the (x, y)-coordinates of the bounding box for
			# the object
			box = detections\[0, 0, i, 3:7\] \* np.array(\[w, h, w, h\])
			(startX, startY, endX, endY) = box.astype("int")

			# update our bounding box results list
			boxes.append((startX, startY, endX, endY))

	# return the face detection bounding boxes
	return boxes

The `detect_faces` function accepts our input face detector `net`, an input `image` to apply face detection to, and the `minConfidence` used to filter out weak/false-positive detections.

We then preprocess the input image such that we can pass it through the face detection model (**Lines 11 and 12**). This function performs resizing, scaling, and mean subtraction.

**Lines 16 and 17** perform face detection, resulting in a `detections` list which we loop over on **Line 21**.

Provided the `confidence` of the detected face is greater than the `minConfidence`, we extract the bounding box coordinates and update our `boxes` list.

The `boxes` list is then returned to the calling function.

Our second function `load_face_dataset`, loops over all images in the CALTECH Faces dataset and applies face detection to each image:

def load\_face\_dataset(inputPath, net, minConfidence=0.5,
	minSamples=15):
	# grab the paths to all images in our input directory, extract
	# the name of the person (i.e., class label) from the directory
	# structure, and count the number of example images we have per
	# face
	imagePaths = list(paths.list\_images(inputPath))
	names = \[p.split(os.path.sep)\[-2\] for p in imagePaths\]
	(names, counts) = np.unique(names, return\_counts=True)
	names = names.tolist()

	# initialize lists to store our extracted faces and associated
	# labels
	faces = \[\]
	labels = \[\]

	# loop over the image paths
	for imagePath in imagePaths:
		# load the image from disk and extract the name of the person
		# from the subdirectory structure
		image = cv2.imread(imagePath)
		name = imagePath.split(os.path.sep)\[-2\]

		# only process images that have a sufficient number of
		# examples belonging to the class
		if counts\[names.index(name)\] < minSamples:
			continue

The `load_face_dataset` requires that we supply the `inputPath` to the CALTECH Faces dataset, the face detection model (`net`), the minimum confidence for a positive detection, and the minimum number of example images required for each face.

**Line 46** grabs the paths to all input images in the CALTECH Faces dataset while **Lines 47-49** extract the names of the individuals from the subdirectory structure, and counts the number of images associated with each person.

We then loop over all `imagePaths` on **Line 57**, loading the `image` from disk and extracting the `name` of the person.

If there are less than `minSamples` images for this particular `name`, we throw out the image and will _not_ consider that individual when training our face recognizer. We do this to avoid class imbalance issues (handling class imbalance is outside the scope of this tutorial).

Provided the `minSamples` test passes we then move on to performing face detection:

		# perform face detection
		boxes = detect\_faces(net, image, minConfidence)

		# loop over the bounding boxes
		for (startX, startY, endX, endY) in boxes:
			# extract the face ROI, resize it, and convert it to
			# grayscale
			faceROI = image\[startY:endY, startX:endX\]
			faceROI = cv2.resize(faceROI, (47, 62))
			faceROI = cv2.cvtColor(faceROI, cv2.COLOR\_BGR2GRAY)

			# update our faces and labels lists
			faces.append(faceROI)
			labels.append(name)

	# convert our faces and labels lists to NumPy arrays
	faces = np.array(faces)
	labels = np.array(labels)

	# return a 2-tuple of the faces and labels
	return (faces, labels)

For each detected face we extract the face ROI, resize it to a fixed size (a _requirement_ when performing PCA), and then convert the image from color to grayscale.

The resulting set of `faces` and `labels` are returned to the calling function.

**_Note:_** _If you would like more details on how both these functions work, I suggest you read my previous guide on_ [_Face Recognition with Local Binary Patterns (LBPs) and OpenCV_](https://pyimagesearch.com/2021/05/03/face-recognition-with-local-binary-patterns-lbps-and-opencv/)_, where I cover both `detect_faces` and `load_face_dataset` in detail._

### **Implementing Eigenfaces with OpenCV**

Let’s now implement Eigenfaces for face recognition with OpenCV!

Open the `eigenfaces.py` file in your project directory structure and let’s get coding:

\# import the necessary packages
from sklearn.preprocessing import LabelEncoder
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.model\_selection import train\_test\_split
from sklearn.metrics import classification\_report
from skimage.exposure import rescale\_intensity
from pyimagesearch.faces import load\_face\_dataset
from imutils import build\_montages
import numpy as np
import argparse
import imutils
import time
import cv2
import os

**Lines 2-15** import our required Python packages. Our notable imports include:

*   `LabelEncoder`: Used to encode the class labels (i.e., names of the individuals) as integers rather than strings (this is a requirement in order to utilize OpenCV’s LBP face recognizer)
*   `PCA`: Performs Principal Component Analysis
*   `SVC`: Our Support Vector Machine (SVM) classifier which we’ll train on top of the eigenface representation of our dataset
*   `train_test_split`: Constructs a training and testing split from our CALTECH Faces dataset
*   `rescale_intensity`: Used to visualize the eigenface representations

We can now move on to command line arguments:

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-i", "--input", type=str, required=True,
	help="path to input directory of images")
ap.add\_argument("-f", "--face", type=str,
	default="face\_detector",
	help="path to face detector model directory")
ap.add\_argument("-c", "--confidence", type=float, default=0.5,
	help="minimum probability to filter weak detections")
ap.add\_argument("-n", "--num-components", type=int, default=150,
	help="# of principal components")
ap.add\_argument("-v", "--visualize", type=int, default=-1,
	help="whether or not PCA components should be visualized")
args = vars(ap.parse\_args())

Our command line arguments include:

1.  `--input`: The path to our input dataset containing images of the individuals we want to train our LBP face recognizer on
2.  `--face`: Path to our OpenCV deep learning face detector
3.  `--confidence`: Minimum probability used to filter out weak detections
4.  `--num-components`: The number of principal components when applying PCA (we’ll be discussing this variable in more detail shortly)
5.  `--visualize`: Whether or not to visualize the Eigenfaces representation of the data

Next, let’s load our face detection model from disk:

\# load our serialized face detector model from disk
print("\[INFO\] loading face detector model...")
prototxtPath = os.path.sep.join(\[args\["face"\], "deploy.prototxt"\])
weightsPath = os.path.sep.join(\[args\["face"\],
	"res10\_300x300\_ssd\_iter\_140000.caffemodel"\])
net = cv2.dnn.readNet(prototxtPath, weightsPath)

And from there, let’s load the CALTECH Faces dataset:

\# load the CALTECH faces dataset
print("\[INFO\] loading dataset...")
(faces, labels) = load\_face\_dataset(args\["input"\], net,
	minConfidence=0.5, minSamples=20)
print("\[INFO\] {} images in dataset".format(len(faces)))

# flatten all 2D faces into a 1D list of pixel intensities
pcaFaces = np.array(\[f.flatten() for f in faces\])

# encode the string labels as integers
le = LabelEncoder()
labels = le.fit\_transform(labels)

# construct our training and testing split
split = train\_test\_split(faces, pcaFaces, labels, test\_size=0.25,
	stratify=labels, random\_state=42)
(origTrain, origTest, trainX, testX, trainY, testY) = split

**Lines 41 and 42** load the CALTECH Faces dataset from disk, resulting a 2-tuple:

1.  `faces`: Face ROIs from the CALTECH Faces dataset
2.  `labels`: The name of the individual in each face ROI

Keep in mind that each face is a 2D, _M×N_ image; however, in order to apply PCA, we need a 1D representation of each face. Therefore, **Line 46** flattens all 2D faces into a 1D list of pixel intensities.

We then encode the `labels` as integers rather than strings.

**Lines 53-55** then construct our training and testing split, using 75% of the data for training and the remaining 25% for evaluation.

Let’s now perform PCA on our 1D lists of faces:

\# compute the PCA (eigenfaces) representation of the data, then
# project the training data onto the eigenfaces subspace
print("\[INFO\] creating eigenfaces...")
pca = PCA(
	svd\_solver="randomized",
	n\_components=args\["num\_components"\],
	whiten=True)
start = time.time()
trainX = pca.fit\_transform(trainX)
end = time.time()
print("\[INFO\] computing eigenfaces took {:.4f} seconds".format(
	end - start))

Here, we indicate _N_, the number of principal components we are going to use when initializing the `PCA` class. After we have found the top `--num-components`, we then use them to project the original training data to the Eigenface subspace.

Now that we’ve performed PCA, let’s visualize the top components:

\# check to see if the PCA components should be visualized
if args\["visualize"\] > 0:
	# initialize the list of images in the montage
	images = \[\]

	# loop over the first 16 individual components
	for (i, component) in enumerate(pca.components\_\[:16\]):
		# reshape the component to a 2D matrix, then convert the data
		# type to an unsigned 8-bit integer so it can be displayed
		# with OpenCV
		component = component.reshape((62, 47))
		component = rescale\_intensity(component, out\_range=(0, 255))
		component = np.dstack(\[component.astype("uint8")\] \* 3)
		images.append(component)

	# construct the montage for the images
	montage = build\_montages(images, (47, 62), (4, 4))\[0\]

	# show the mean and principal component visualizations
	# show the mean image
	mean = pca.mean\_.reshape((62, 47))
	mean = rescale\_intensity(mean, out\_range=(0, 255)).astype("uint8")
	cv2.imshow("Mean", mean)
	cv2.imshow("Components", montage)
	cv2.waitKey(0)

**Line 71** makes a check to see if the `--visualize` command line argument is set, and if so, we initialize a list of `images` to store our visualizations.

From there, we loop over each of the top PCA components (**Line 76**), reshape the image into a _47×62_ pixel bitmap image (**Line 80**), and then rescale the pixel intensities to the range _\[0, 255\]_ **(Line 81**)_._

Why do we bother with the rescaling operation?

Simple — our eigenvalue decomposition results in real-valued feature vectors, but in order to visualize images with OpenCV and `cv2.imshow`, our images must be unsigned 8-bit integers in the range _\[0, 255\] —_ **Lines 81 and 82** take care of that operation for us.

The resulting `component` is then added to our list of `images` for visualization.

On **Line 86** we build a `montage` of the top components.

We then display the mean eigenvector representation in similar fashion on **Lines 90-94.**

With visualization taken care of, let’s train our SVM on the eigenface representations:

\# train a classifier on the eigenfaces representation
print("\[INFO\] training classifier...")
model = SVC(kernel="rbf", C=10.0, gamma=0.001, random\_state=42)
model.fit(trainX, trainY)

# evaluate the model
print("\[INFO\] evaluating model...")
predictions = model.predict(pca.transform(testX))
print(classification\_report(testY, predictions,
	target\_names=le.classes\_))

**Lines 98 and 99** initialize our SVM and train it.

We then use the model to make predictions on our testing data, taking care to project the testing data onto the eigenvalue subspace _before_ making predictions.

**This projection is a** **_requirement_****. If you forget to perform the projection, one of two things will happen:**

1.  Your code will error out (due to a dimensionality mismatch between the feature vectors and the SVM model)
2.  The SVM will return nonsense classifications (because the data the model was trained on was projected onto the eigenfaces representation)

**Lines 104 and 105** then show a classification report, displaying the accuracy of our Eigenfaces recognition model.

The final step is to sample our testing data, make predictions on it, and display the results individually to our screen:

\# generate a sample of testing data
idxs = np.random.choice(range(0, len(testY)), size=10, replace=False)

# loop over a sample of the testing data
for i in idxs:
	# grab the predicted name and actual name
	predName = le.inverse\_transform(\[predictions\[i\]\])\[0\]
	actualName = le.classes\_\[testY\[i\]\]

	# grab the face image and resize it such that we can easily see
	# it on our screen
	face = np.dstack(\[origTest\[i\]\] \* 3)
	face = imutils.resize(face, width=250)

	# draw the predicted name and actual name on the image
	cv2.putText(face, "pred: {}".format(predName), (5, 25),
		cv2.FONT\_HERSHEY\_SIMPLEX, 0.8, (0, 255, 0), 2)
	cv2.putText(face, "actual: {}".format(actualName), (5, 60),
		cv2.FONT\_HERSHEY\_SIMPLEX, 0.8, (0, 0, 255), 2)

	# display the predicted name  and actual name
	print("\[INFO\] prediction: {}, actual: {}".format(
		predName, actualName))

	# display the current face to our screen
	cv2.imshow("Face", face)
	cv2.waitKey(0)

We covered this code block in detail in last week’s guide on [LBP face recognition](https://pyimagesearch.com/2021/05/03/face-recognition-with-local-binary-patterns-lbps-and-opencv/) so be sure to refer to it if you would like more details.

### **OpenCV Eigenfaces face recognition results**

We are now ready to recognize faces using OpenCV and the Eigenfaces algorithm!

Start by accessing the **_“Downloads”_** section of this tutorial to retrieve the source code and CALTECH Faces dataset.

From there, open a terminal and execute the following command:

$ python eigenfaces.py --input caltech\_faces --visualize 1
\[INFO\] loading face detector model...
\[INFO\] loading dataset...
\[INFO\] 397 images in dataset
\[INFO\] creating eigenfaces...
\[INFO\] computing eigenfaces took 0.1049 seconds
\[INFO\] training classifier...
\[INFO\] evaluating model...
              precision    recall  f1-score   support

     abraham       1.00      1.00      1.00         5
       allen       1.00      0.75      0.86         8
      carmen       1.00      1.00      1.00         5
      conrad       0.86      1.00      0.92         6
     cynthia       1.00      1.00      1.00         5
     darrell       1.00      1.00      1.00         5
       frank       0.83      1.00      0.91         5
      gloria       1.00      1.00      1.00         5
     jacques       0.86      1.00      0.92         6
        judy       1.00      1.00      1.00         5
       julie       1.00      1.00      1.00         5
     kenneth       1.00      1.00      1.00         6
         mae       1.00      1.00      1.00         5
     raymond       1.00      1.00      1.00         6
        rick       1.00      1.00      1.00         6
      sherry       1.00      0.83      0.91         6
     tiffany       1.00      1.00      1.00         5
      willie       1.00      1.00      1.00         6

    accuracy                           0.97       100
   macro avg       0.97      0.98      0.97       100
weighted avg       0.97      0.97      0.97       100

Once our face images are loaded from disk, computing the Eigenfaces representation (i.e., eigenvalue decomposition) is quite fast, taking just over a tenth of a second.

Given the Eigenfaces, we then train our SVM. Evaluated on our testing set, we see we obtain ****≈**97% accuracy**. Not too bad!

Let’s now move on to identifying faces in individual images:

\[INFO\] prediction: frank, actual: frank
\[INFO\] prediction: abraham, actual: abraham
\[INFO\] prediction: julie, actual: julie
\[INFO\] prediction: tiffany, actual: tiffany
\[INFO\] prediction: julie, actual: julie
\[INFO\] prediction: mae, actual: mae
\[INFO\] prediction: allen, actual: allen
\[INFO\] prediction: mae, actual: mae
\[INFO\] prediction: conrad, actual: conrad
\[INFO\] prediction: darrell, actual: darrell

![Image 22](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20461%201024'%3E%3C/svg%3E)

**Figure 10:** A sample of our face recognition results using Eigenfaces and OpenCV.

**Figure 10** displays a montage of face recognition results using Eigenfaces. Notice how we’ve been able to correctly identify each of the faces.

### **Problems with Eigenfaces for face recognition**

One of the biggest criticisms of the Eigenfaces algorithm is the strict [facial alignment](https://pyimagesearch.com/2017/05/22/face-alignment-with-opencv-and-python/) required when training and identifying faces:

![Image 23](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20341'%3E%3C/svg%3E)

**Figure 11:** Performing face alignment can improve your face recognition accuracy.

**Since we are operating at the pixel level, facial features such as eyes, nose, and mouth need to be aligned** **_near perfectly_** **for all images in our dataset.** Not only is this a challenging task, it can be very hard to guarantee in real-world situations where conditions are less than ideal.

In our case, most faces in the CALTECH Faces dataset were captured from a frontal view with no viewpoint change, head tilt, etc. Due to this face we didn’t need to explicitly apply face alignment and everything worked out in our favor; however, it’s worth noting that this situation _rarely_ happens in real-world conditions.

Last week we discussed [Local Binary Patterns (LBPs) for face recognition](https://pyimagesearch.com/2021/05/03/face-recognition-with-local-binary-patterns-lbps-and-opencv/). While LBPs can also suffer from not having adequate facial alignment, the results are far less pronounced and the approach is much more robust, since we split the face image into _7×7_ cells and compute an LBP histogram for each cell

By quantifying the image into concatenated feature vectors, we are normally able to add additional accuracy to our face identifier — because of the added robustness, we normally use LBPs for face recognition rather than the original Eigenfaces algorithm.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial, we discussed the Eigenfaces algorithm.

The first step in Eigenfaces is to supply a dataset of images with multiple images per person you want to recognize.

From there, we flatten each image into a vector and store them in a matrix. Principal Component Analysis is applied to this matrix, where we take the top _N_ eigenvectors with the corresponding largest eigenvalue magnitude — these _N_ eigenvectors are our eigenfaces.

For many machine learning algorithms, it’s often quite hard to visualize the results of a particular method. But for Eigenfaces, it’s actually quite easy! Each eigenface is simply a ![Image 24: K^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2021%2014'%3E%3C/svg%3E) flattened vector that can be reshaped into a ![Image 25: K \times K](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2050%2011'%3E%3C/svg%3E) image, allowing us to visualize each eigenface.

To identify a face, we first project the image onto the eigenface subspace, followed by utilizing a k-NN classifier with the Euclidean distance to find the nearest neighbor(s). These nearest neighbors are used to determine the overall identification. However, we can improve accuracy further by using more advanced machine learning algorithms, which is exactly what we did in this lesson.

Last week we discussed [Local Binary Patterns (LBPs) for face recognition](https://pyimagesearch.com/2021/05/03/face-recognition-with-local-binary-patterns-lbps-and-opencv/). In practice, this method tends to be a bit more robust than Eigenfaces, obtaining higher face recognition accuracy.

While neither method is as accurate as our modern deep learning face recognition models, it’s still important to understand from a historical perspective, and when applying deep learning models is just not computationally feasible.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch),** **_simply enter your email address in the form below!_**

![Image 26](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!