Title: Smile detection with OpenCV, Keras, and TensorFlow - PyImageSearch

URL Source: https://pyimagesearch.com/2021/07/14/smile-detection-with-opencv-keras-and-tensorflow/

Published Time: 2021-07-14T14:00:00+00:00

Markdown Content:
In this tutorial, we will be building a complete end-to-end application that can detect smiles in a video stream in real-time using deep learning along with traditional computer vision techniques.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20650%20365'%3E%3C/svg%3E)

To accomplish this task, we’ll be training the LetNet architecture on a dataset of images that contain faces of people who are _smiling_ and _not smiling_. Once our network is trained, we’ll create a separate Python script — this one will detect faces in images via OpenCV’s built-in Haar cascade face detector, extract the face region of interest (ROI) from the image, and then pass the ROI through LeNet for smile detection.

**To learn how to detect a smile with OpenCV, Keras, and TensorFlow,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

When developing real-world applications for image classification, you’ll often have to mix traditional computer vision and image processing techniques with deep learning. I’ve done my best to ensure this tutorial stands on its own in terms of algorithms, techniques, and libraries you need to understand in order to be successful when studying and applying deep learning.

### **Configuring your development environment**

To follow this guide, you need to have the OpenCV library installed on your system.

Luckily, OpenCV is pip-installable:

$ pip install opencv-contrib-python

**If you need help configuring your development environment for OpenCV, I _highly recommend_ that you read my** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in a matter of minutes.

### **Having problems configuring your development environment?**

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 1:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

### **The SMILES Dataset**

The SMILES dataset consists of images of faces that are either _smiling_ or _not smiling_ ([Hromada, 2010](https://github.com/hromi/SMILEsmileD)). In total, there are 13,165 grayscale images in the dataset, with each image having a size of 64_×_64 pixels.

As **Figure 2** demonstrates, images in this dataset are _tightly cropped_ around the face, which will make the training process easier as we’ll be able to learn the “smiling” or “not smiling” patterns directly from the input images.

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20276'%3E%3C/svg%3E)

**Figure 2:** _Top:_ Examples of “smiling” faces. _Bottom:_ Samples of “not smiling” faces. In this tutorial, we will be training a Convolutional Neural Network to recognize between smiling and not smiling faces in real-time video streams.

However, the close cropping poses a problem during testing — since our input images will not only contain a face but the _background_ of the image as well, we first need to _localize_ the face in the image and extract the face ROI before we can pass it through our network for detection. Luckily, using traditional computer vision methods such as Haar cascades, this is a much easier task than it sounds.

A second issue we need to handle in the SMILES dataset is _class imbalance_. While there are 13,165 images in the dataset, 9,475 of these examples are _not smiling,_ while only 3,690 belong to the _smiling_ class. Given that there are over 2_._5_x_ the number of “not smiling” images to “smiling” examples, we need to be careful when devising our training procedure.

Our network may _naturally_ pick the “not smiling” label since (1) the distributions are uneven and (2) it has more examples of what a “not smiling” face looks like. Later, you will see how we can combat class imbalance by computing a “weight” for each class during training time.

### **Training the Smile CNN**

The first step in building our smile detector is to train a CNN on the SMILES dataset to distinguish between a face that is smiling versus not smiling. To accomplish this task, let’s create a new file named `train_model.py`. From there, insert the following code:

\# import the necessary packages
from sklearn.preprocessing import LabelEncoder
from sklearn.model\_selection import train\_test\_split
from sklearn.metrics import classification\_report
from tensorflow.keras.preprocessing.image import img\_to\_array
from tensorflow.keras.utils import to\_categorical
from pyimagesearch.nn.conv import LeNet
from imutils import paths
import matplotlib.pyplot as plt
import numpy as np
import argparse
import imutils
import cv2
import os

**Lines 2-14** import our required Python packages. We’ve used all of the packages before, but I want to call your attention to **Line 7**, where we import the LeNet **(**[LeNet Tutorial](https://pyimagesearch.com/2021/05/22/lenet-recognizing-handwritten-digits/)) class — this is the architecture we’ll be using when creating our smile detector.

Next, let’s parse our command line arguments:

\# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-d", "--dataset", required=True,
	help="path to input dataset of faces")
ap.add\_argument("-m", "--model", required=True,
	help="path to output model")
args = vars(ap.parse\_args())

# initialize the list of data and labels
data = \[\]
labels = \[\]

Our script will require two command line arguments, each of which I’ve detailed below:

1.  `--dataset`: The path to the SMILES directory residing on disk.
2.  `--model`: The path to where the serialized LeNet weights will be saved after training.

We are now ready to load the SMILES dataset from disk and store it in memory:

\# loop over the input images
for imagePath in sorted(list(paths.list\_images(args\["dataset"\]))):
	# load the image, pre-process it, and store it in the data list
	image = cv2.imread(imagePath)
	image = cv2.cvtColor(image, cv2.COLOR\_BGR2GRAY)
	image = imutils.resize(image, width=28)
	image = img\_to\_array(image)
	data.append(image)

	# extract the class label from the image path and update the
	# labels list
	label = imagePath.split(os.path.sep)\[-3\]
	label = "smiling" if label == "positives" else "not\_smiling"
	labels.append(label)

On **Line 29,** we loop over all images in the `--dataset` input directory. For each of these images, we:

1.  Load it from disk (**Line 31**).
2.  Convert it to grayscale (**Line 32**).
3.  Resize it to have a _fixed input size_ of `28×28` pixels (**Line 33**).
4.  Convert the image to an array compatible with Keras and its channel ordering (**Line 34**).
5.  Add the `image` to the `data` list that LeNet will be trained on.

**Lines 39-41** handle extracting the class label from the `imagePath` and updating the `labels` list. The SMILES dataset stores _smiling_ faces in the `SMILES/positives/positives7` subdirectory, while _not smiling_ faces live in the `SMILES/negatives/negatives7` subdirectory.

Therefore, given the path to an image:

SMILEs/positives/positives7/10007.jpg

We can extract the class label by splitting on the image path separator and grabbing the third-to-last subdirectory: `positives`. In fact, this is exactly what **Line 39** accomplishes.

Now that our `data` and `labels` are constructed, we can scale the raw pixel intensities to the range `[0, 1]` and then apply one-hot encoding to the `labels`:

\# scale the raw pixel intensities to the range \[0, 1\]
data = np.array(data, dtype="float") / 255.0
labels = np.array(labels)

# convert the labels from integers to vectors
le = LabelEncoder().fit(labels)
labels = to\_categorical(le.transform(labels), 2)

Our next code block handles our data imbalance issue by computing the class weights:

\# calculate the total number of training images in each class and
# initialize a dictionary to store the class weights
classTotals = labels.sum(axis=0)
classWeight = dict()

# loop over all classes and calculate the class weight
for i in range(0, len(classTotals)):
	classWeight\[i\] = classTotals.max() / classTotals\[i\]

**Line 53** computes the total number of examples per class. In this case, `classTotals` will be an array: `[9475, 3690]` for “not smiling” and “smiling,” respectively.

We then _scale_ these totals on **Lines 57 and 58** to obtain the `classWeight` used to handle the class imbalance, yielding the array: `[1, 2.56]`. This weighting implies that our network will treat every instance of “smiling” as 2.56 instances of “not smiling” and helps combat the class imbalance issue by amplifying the per-instance loss by a larger weight when seeing “smiling” examples.

Now that we’ve computed our class weights, we can move on to partitioning our data into training and testing splits, using 80% of the data for training and 20% for testing:

\# partition the data into training and testing splits using 80% of
# the data for training and the remaining 20% for testing
(trainX, testX, trainY, testY) = train\_test\_split(data,
	labels, test\_size=0.20, stratify=labels, random\_state=42)

Finally, we are ready to train LeNet:

\# initialize the model
print("\[INFO\] compiling model...")
model = LeNet.build(width=28, height=28, depth=1, classes=2)
model.compile(loss="binary\_crossentropy", optimizer="adam",
	metrics=\["accuracy"\])

# train the network
print("\[INFO\] training network...")
H = model.fit(trainX, trainY, validation\_data=(testX, testY),
	class\_weight=classWeight, batch\_size=64, epochs=15, verbose=1)

**Line 67** initializes the LeNet architecture that will accept `28×28` single channel images. Given that there are only two classes (smiling versus not smiling), we set `classes=2`.

We’ll also be using `binary_crossentropy` rather than `categorical_crossentropy` as our loss function. Again, categorical cross-entropy is only used when the number of classes is more than two.

Up until this point, we’ve been using the SGD optimizer to train our network. Here, we’ll be using Adam ([Kingma and Ba, 2014](http://arxiv.org/abs/1412.6980)) (**Line 68**).

Again, the optimizer and associated parameters are often considered hyperparameters that you need to tune when training your network. When I put this example together, I found that Adam performed substantially better than SGD.

**Lines 73 and 74** train LeNet for a total of 15 epochs using our supplied `classWeight` to combat class imbalance.

Once our network is trained, we can evaluate it and serialize the weights to disk:

\# evaluate the network
print("\[INFO\] evaluating network...")
predictions = model.predict(testX, batch\_size=64)
print(classification\_report(testY.argmax(axis=1),
	predictions.argmax(axis=1), target\_names=le.classes\_))

# save the model to disk
print("\[INFO\] serializing network...")
model.save(args\["model"\])

We’ll also construct a learning curve for our network so we can visualize performance:

\# plot the training + testing loss and accuracy
plt.style.use("ggplot")
plt.figure()
plt.plot(np.arange(0, 15), H.history\["loss"\], label="train\_loss")
plt.plot(np.arange(0, 15), H.history\["val\_loss"\], label="val\_loss")
plt.plot(np.arange(0, 15), H.history\["accuracy"\], label="acc")
plt.plot(np.arange(0, 15), H.history\["val\_accuracy"\], label="val\_acc")
plt.title("Training Loss and Accuracy")
plt.xlabel("Epoch #")
plt.ylabel("Loss/Accuracy")
plt.legend()
plt.show()

To train our smile detector, execute the following command:

$ python train\_model.py --dataset ../datasets/SMILEsmileD \\
	--model output/lenet.hdf5
\[INFO\] compiling model...
\[INFO\] training network...
Train on 10532 samples, validate on 2633 samples
Epoch 1/15
8s - loss: 0.3970 - acc: 0.8161 - val\_loss: 0.2771 - val\_acc: 0.8872
Epoch 2/15
8s - loss: 0.2572 - acc: 0.8919 - val\_loss: 0.2620 - val\_acc: 0.8899
Epoch 3/15
7s - loss: 0.2322 - acc: 0.9079 - val\_loss: 0.2433 - val\_acc: 0.9062
...
Epoch 15/15
8s - loss: 0.0791 - acc: 0.9716 - val\_loss: 0.2148 - val\_acc: 0.9351
\[INFO\] evaluating network...
             precision    recall  f1-score   support

not\_smiling       0.95      0.97      0.96      1890
    smiling       0.91      0.86      0.88       743

avg / total       0.93      0.94      0.93      2633

\[INFO\] serializing network...

After 15 epochs, we can see that our network is obtaining 93% classification accuracy. **Figure 3** plots our learning curve:

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20800%20600'%3E%3C/svg%3E)

**Figure 3:** A plot of the learning curve for the LeNet architecture trained on the SMILES dataset. After fifteen epochs we are obtaining _≈_93% classification accuracy on our testing set.

Past epoch six our validation loss starts to stagnate — further training past epoch 15 would result in overfitting. If desired, we would improve the accuracy of our smile detector by using more training data, either by:

1.  Gathering additional training data.
2.  Applying _data augmentation_ to randomly translate, rotate, and shift our _existing_ training set.

### **Running the Smile CNN in Real-time**

Now that we’ve trained our model, the next step is to build the Python script to access our webcam/video file and apply smile detection to each frame. To accomplish this step, open a new file, name it `detect_smile.py`, and we’ll get to work.

\# import the necessary packages
from tensorflow.keras.preprocessing.image import img\_to\_array
from tensorflow.keras.models import load\_model
import numpy as np
import argparse
import imutils
import cv2

**Lines 2-7** import our required Python packages. The `img_to_array` function will be used to convert each individual frame from our video stream to a properly channel ordered array. The `load_model` function will be used to load the weights of our trained LeNet model from disk.

The `detect_smile.py` script requires two command line arguments followed by a third optional one:

\# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-c", "--cascade", required=True,
	help="path to where the face cascade resides")
ap.add\_argument("-m", "--model", required=True,
	help="path to pre-trained smile detector CNN")
ap.add\_argument("-v", "--video",
	help="path to the (optional) video file")
args = vars(ap.parse\_args())

The first argument, `--cascade` is the path to a Haar cascade used to detect faces in images. First published in 2001, Paul Viola and Michael Jones detail the Haar cascade in their work, _[Rapid Object Detection using a Boosted Cascade of Simple Features](https://ieeexplore.ieee.org/document/990517)_. This publication has become one of the most cited papers in the computer vision literature.

The Haar cascade algorithm is capable of detecting objects in images, regardless of their location and scale. Perhaps most intriguing (and relevant to our application), the detector can run in real-time on modern hardware. In fact, the motivation behind Viola and Jones’ work was to create a _face detector_.

The second common line argument, `--model`, specifies the path to our serialized LeNet weights on disk. Our script will _default_ to reading frames from a built-in/USB webcam; however, if we instead want to read frames from a file, we can specify the file via the optional `--video` switch.

Before we can detect smiles, we first need to perform some initializations:

\# load the face detector cascade and smile detector CNN
detector = cv2.CascadeClassifier(args\["cascade"\])
model = load\_model(args\["model"\])

# if a video path was not supplied, grab the reference to the webcam
if not args.get("video", False):
	camera = cv2.VideoCapture(0)

# otherwise, load the video
else:
	camera = cv2.VideoCapture(args\["video"\])

**Lines 20 and 21** load the Haar cascade face detector and the pre-trained LeNet model, respectively. If a video path was _not_ supplied, we grab a pointer to our webcam (**Lines 24 and 25**). Otherwise, we open a pointer to the video file on disk (**Lines 28 and 29**).

We have now reached the main processing pipeline of our application:

\# keep looping
while True:
	# grab the current frame
	(grabbed, frame) = camera.read()

	# if we are viewing a video and we did not grab a frame, then we
	# have reached the end of the video
	if args.get("video") and not grabbed:
		break

	# resize the frame, convert it to grayscale, and then clone the
	# original frame so we can draw on it later in the program
	frame = imutils.resize(frame, width=300)
	gray = cv2.cvtColor(frame, cv2.COLOR\_BGR2GRAY)
	frameClone = frame.copy()

**Line 32** starts a loop that will continue until (1) we stop the script or (2) we reach the end of the video file (provided a `--video` path was applied).

**Line 34** grabs the next frame from the video stream. If the `frame` could not be `grabbed`, then we have reached the end of the video file. Otherwise, we pre-process the `frame` for face detection by resizing it to have a width of 300 pixels (**Line 43**) and converting it to grayscale (**Line 44**).

The `.detectMultiScale` method handles detecting the bounding box (_x, y_)-coordinates of faces in the `frame`:

	# detect faces in the input frame, then clone the frame so that
	# we can draw on it
	rects = detector.detectMultiScale(gray, scaleFactor=1.1, 
		minNeighbors=5, minSize=(30, 30),
		flags=cv2.CASCADE\_SCALE\_IMAGE)

Here, we pass in our grayscale image and indicate that for a given region to be considered a face it _must_ have a minimum width of `30×30` pixels. The `minNeighbors` attribute helps prune false positives while the `scaleFactor` controls the number of image pyramid ([http://pyimg.co/rtped](http://pyimg.co/rtped)) levels generated.

Again, a detailed review of Haar cascades for object detection is outside the scope of this tutorial.

The `.detectMultiScale` method returns a list of 4-tuples that make up the _rectangle_ that bounds the face in the `frame`. The first two values in this list are the starting (_x, y_)-coordinates. The second two values in the `rects` list are the width and height of the bounding box, respectively.

We loop over each set of bounding boxes below:

	# loop over the face bounding boxes
	for (fX, fY, fW, fH) in rects:
		# extract the ROI of the face from the grayscale image,
		# resize it to a fixed 28x28 pixels, and then prepare the
		# ROI for classification via the CNN
		roi = gray\[fY:fY + fH, fX:fX + fW\]
		roi = cv2.resize(roi, (28, 28))
		roi = roi.astype("float") / 255.0
		roi = img\_to\_array(roi)
		roi = np.expand\_dims(roi, axis=0)

For each of the bounding boxes, we use NumPy array slicing to extract the face ROI (**Line 58**). Once we have the ROI, we preprocess it and prepare it for classification via LeNet by resizing it, scaling it, converting it to a Keras-compatible array, and padding the image with an extra dimension (**Lines 59-62**).

Once the `roi` is preprocessed, it can be passed through LeNet for classification:

		# determine the probabilities of both "smiling" and "not
		# smiling", then set the label accordingly
		(notSmiling, smiling) = model.predict(roi)\[0\]
		label = "Smiling" if smiling > notSmiling else "Not Smiling"

A call to `.predict` on **Line 66** returns the _probabilities_ of “not smiling” and “smiling,” respectively. **Line 67** sets the `label` depending on which probability is larger.

Once we have the `label`, we can draw it, along with the corresponding bounding box on the frame:

		# display the label and bounding box rectangle on the output
		# frame
		cv2.putText(frameClone, label, (fX, fY - 10),
			cv2.FONT\_HERSHEY\_SIMPLEX, 0.45, (0, 0, 255), 2)
		cv2.rectangle(frameClone, (fX, fY), (fX + fW, fY + fH),
			(0, 0, 255), 2)

Our final code block handles displaying the output frame on our screen:

	# show our detected faces along with smiling/not smiling labels
	cv2.imshow("Face", frameClone)

	# if the 'q' key is pressed, stop the loop
	if cv2.waitKey(1) & 0xFF == ord("q"):
		break

# cleanup the camera and close any open windows
camera.release()
cv2.destroyAllWindows()

If the `q` key is pressed, we exit the script.

To run `detect_smile.py` using your webcam, execute the following command:

$ python detect\_smile.py --cascade haarcascade\_frontalface\_default.xml \\
	--model output/lenet.hdf5 

If you instead want to use a video file, you would update your command to use the `--video` switch:

$ python detect\_smile.py --cascade haarcascade\_frontalface\_default.xml \\
	--model output/lenet.hdf5 --video path/to/your/video.mov

I have included the results of the smile detection script in **Figure 4**:

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20379%20500'%3E%3C/svg%3E)

**Figure 4:** Applying our CNN to recognize smiling vs. not-smiling in real-time video streams on a CPU.

Notice how LeNet is correctly predicting “smiling” or “not smiling” based on my facial expression.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial, we learned how to build an end-to-end computer vision and deep learning application to perform smile detection. To do so, we first trained the LeNet architecture on the SMILES dataset. Due to class imbalances in the SMILES dataset, we discovered how to compute class weights used to help mitigate the problem.

Once trained, we evaluated LeNet on our testing set and found the network obtained a respectable 93% classification accuracy. Higher classification accuracy can be obtained by gathering more training data or applying data augmentation to _existing_ training data.

We then created a Python script to read frames from a webcam/video file, detect faces, and then apply our pre-trained network. To detect faces, we used OpenCV’s Haar cascades. Once a face was detected it was extracted from the frame and then passed through LeNet to determine if the person was smiling or not smiling. As a whole, our smile detection system can easily run in real-time on the CPU using modern hardware.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!