Computer Vision — Detecting objects using Haar Cascade Classifier | by Tanwir Khan | Towards Data Science

Before we start getting into details, let’s first understand what is computer vision and some of its applications.
Computer vision is a field of study which encompasses on how computer see and understand digital images and videos.
Computer vision involves seeing or sensing a visual stimulus, make sense of what it has seen and also extract complex information that could be used for other machine learning activities.
There are many practical applications of computer vision:
Autonomous Vehicles — This is one of the most important applications of Computer vision where the self-driving cars need to gather information about their surroundings to decide how to behave.
Facial Recognition — This is also a very important application of computer vision where electronics use facial recognition technology to basically validate the identity of the user.
Image Search and Object Recognition — Now we could search objects in an image using image search. A very good example is google lens where we could search a particular object within the image by clicking the photo of the image and the computer vision algorithm will search through the catalogue of images and extract information out of the image.
Robotics — Most robotic machines, often in manufacturing, need to see their surroundings to perform the task at hand. In manufacturing machines may be used to inspect assembly tolerances by “looking at” them.
Now that we know what does Computer Vision mean and some of its application, let’s dive into the implementation of it. To implement various examples of computer vision, we are going to use the OpenCV library.
OpenCV (Open Source Computer Vision Library: http://opencv.org) is an open-source BSD-licensed library that includes several hundreds of computer vision algorithms.
Haar Cascade Classifiers : We will implement our use case using the Haar Cascade classifier. Haar Cascade classifier is an effective object detection approach which was proposed by Paul Viola and Michael Jones in their paper, “Rapid Object Detection using a Boosted Cascade of Simple Features” in 2001.
So, let’s try to understand what these Haar Cascade Classifiers are.This is basically a machine learning based approach where a cascade function is trained from a lot of images both positive and negative. Based on the training it is then used to detect the objects in the other images.
So how this works is they are huge individual .xml files with a lot of feature sets and each xml corresponds to a very specific type of use case.
For Example, if you go to the github page of haarcascade you will see that there is a particular xml file containing the feature set to detect the full body, lower-body, eye, frontal-face and so on.
To understand how the haar cascade classifier work and how will that be used for computer vision, let’s implement some of the use cases.
In this use-case we will try to detect the face of individuals using the haarcascade_frontalface_default.xml
The dimensions of the image that we have used here was pretty large, so we have scaled down the image dimensions for better output.
We resize the image using the below code:
Now that we have resized our image, let’s go ahead and import cv2 and numpy and also use the CascadeClassifier function of OpenCV to point to the location where we have stored the XML file, haarcascade_frontalface_default.xml in our case. I have downloaded the xml file to my local and used the path of my machine, but you could directly point it to the github location if you want.
import cv2import numpy as npface_classifier = cv2.CascadeClassifier('/haarcascade_frontalface_default.xml')
Now the 2nd step is to load the image and convert it into gray-scale. Before showing the code, I want to tell you the reason why we are converting the image to grayscale here.
Generally the images that we see are in the form of RGB channel(Red, Green, Blue). So, when OpenCV reads the RGB image, it usually stores the image in BGR (Blue, Green, Red) channel. For the purposes of image recognition, we need to convert this BGR channel to gray channel. The reason for this is gray channel is easy to process and is computationally less intensive as it contains only 1-channel of black-white.
gray = cv2.cvtColor(resized, cv2.COLOR_BGR2GRAY)
Here the parameters to the function cvtColor will be the image variable name (resized in our case) and the COLOR_BGR2GRAY.
Now after converting the image from RGB to Gray, we will now try to locate the exact features in our face. Let’s see how we could implement that in code.
faces = face_classifier.detectMultiScale(gray, 1.0485258, 6)
In this piece of code what we are trying to do is, using the face_classifier which is an object loaded with haarcascade_frontalface_default.xml, we are using an inbuilt function with it called the detectMultiScale.
This function will help us to find the features/locations of the new image. The way it does is, it will use all the features from the face_classifier object to detect the features of the new image.
The parameters that we will pass to this function are:
The gray scale variable — gray in our casescaleFactor — Parameter specifying how much the image size is reduced  at each image scale. Basically, the scale factor is used to create your scale pyramid. More explanation, your model has a fixed size defined during training, which is visible in the XML. This means that this size of the face is detected in the image if present. However, by rescaling the input image, you can resize a larger face to a smaller one, making it detectable by the algorithm. 1.05 is a good possible value for this, which means you use a small step for resizing, i.e. reduce the size by 5%, you increase the chance of a matching size with the model for detection is found. This also means that the algorithm works slower since it is more thorough. You may increase it to as much as 1.4 for faster detection, with the risk of missing some faces altogether. In our case, I have used 1.0485258 as the scaleFactor as this worked perfectly for the image that I was using.minNeighbors — Parameter specifying how many neighbors each candidate rectangle should have to retain it. This parameter will affect the quality of the detected faces. Higher value results in fewer detections but with higher quality. 3~6 is a good value for it. In our case, I have taken 6 as the minNeighbors and this has worked perfectly for the image that I have used.
The gray scale variable — gray in our case
scaleFactor — Parameter specifying how much the image size is reduced  at each image scale. Basically, the scale factor is used to create your scale pyramid. More explanation, your model has a fixed size defined during training, which is visible in the XML. This means that this size of the face is detected in the image if present. However, by rescaling the input image, you can resize a larger face to a smaller one, making it detectable by the algorithm. 1.05 is a good possible value for this, which means you use a small step for resizing, i.e. reduce the size by 5%, you increase the chance of a matching size with the model for detection is found. This also means that the algorithm works slower since it is more thorough. You may increase it to as much as 1.4 for faster detection, with the risk of missing some faces altogether. In our case, I have used 1.0485258 as the scaleFactor as this worked perfectly for the image that I was using.
minNeighbors — Parameter specifying how many neighbors each candidate rectangle should have to retain it. This parameter will affect the quality of the detected faces. Higher value results in fewer detections but with higher quality. 3~6 is a good value for it. In our case, I have taken 6 as the minNeighbors and this has worked perfectly for the image that I have used.
From the above step, the function detectMultiScale returns 4 values — x-coordinate, y-coordinate, width(w) and height(h) of the detected feature of the face. Based on these 4 values we will draw a rectangle around the face.
if faces is ():    print("No faces found")for (x,y,w,h) in faces:    cv2.rectangle(resized, (x,y), (x+w,y+h), (127,0,255), 2)    cv2.imshow('Face Detection', resized)    cv2.waitKey(0)    cv2.destroyAllWindows()
Consolidating all the 4 steps together, below is the code for it:
import cv2import numpy as npface_classifier = cv2.CascadeClassifier('/haarcascade_frontalface_default.xml')gray = cv2.cvtColor(resized, cv2.COLOR_BGR2GRAY)''' Our classifier returns the ROI of the detected face as a tuple, It stores the top left coordinate and the bottom right coordiantes'''faces = face_classifier.detectMultiScale(gray, 1.0485258, 6)'''When no faces detected, face_classifier returns and empty tuple'''if faces is ():    print("No faces found")'''We iterate through our faces array and draw a rectangle over each face in faces'''for (x,y,w,h) in faces:    cv2.rectangle(resized, (x,y), (x+w,y+h), (127,0,255), 2)    cv2.imshow('Face Detection', resized)    cv2.waitKey(0)    cv2.destroyAllWindows()
I have created the output of the above program as a video and embedded it below. If you see, the program is not able to properly detect some faces. The reason is for haarcascade classifier to work properly, the face should be properly visible. The one which has not been detected in the output are somewhat not clear.
Nonetheless, I will try to improve this in my future posts where we will use different types of algorithms involving deep learning to mitigate this.
Now that we have some working knowledge on how face detection works, let’s see how we could detect both face and Eye
In this implementation, there are couple of changes that we have made in comparison to the Face detection code.
In order to detect the eye we have also imported the haarcascade_eye.xml file.
eye_classifier = cv2.CascadeClassifier('/haarcascade_eye.xml')
Once we get the x-coordinate, y-coordinate, width (w) and height(h) of the detected feature of face using the detectMultiScale function, we will create 2 numpy array — roi_gray and roi_color. The roi_gray is the numpy array created using the variable “gray”. The reason we created this is to pass this gray channel version to the detectMultiScale function while extracting the features (ex,ey,ew,eh) of the eyes.
for (x,y,w,h) in faces:    cv2.rectangle(img,(x,y),(x+w,y+h),(127,0,255),2)    cv2.imshow('img',img)    cv2.waitKey(0)    roi_gray = gray[y:y+h, x:x+w]    roi_color = img[y:y+h, x:x+w]    eyes = eye_classifier.detectMultiScale(roi_gray)
Once we have extracted the features of the eye (ex,ey,ew,eh) we will loop through them and create a rectangle by passing the numpy array roi_color. The reason behind passing roi_color and not roi_gray is roi_color is the array for the actual RGB scale image whereas roi_gray is the gray scale of the image which we have used in our code for faster processing while extracting the dimensions/coordinates of the image and then use those dimensions to pass it in the original array which is roi_color in our case.
for (ex,ey,ew,eh) in eyes:        cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(255,255,0),2)
Consolidating all the steps together:
import numpy as npimport cv2 face_classifier = cv2.CascadeClassifier('/haarcascade_frontalface_default.xml')eye_classifier = cv2.CascadeClassifier('/haarcascade_eye.xml')img = cv2.imread('/group.jpg')gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)faces = face_classifier.detectMultiScale(gray, 1.05, 3)# When no faces detected, face_classifier returns and empty tupleif faces is ():    print("No Face Found")for (x,y,w,h) in faces:    cv2.rectangle(img,(x,y),(x+w,y+h),(127,0,255),2)    cv2.imshow('img',img)    cv2.waitKey(0)    roi_gray = gray[y:y+h, x:x+w]    roi_color = img[y:y+h, x:x+w]    eyes = eye_classifier.detectMultiScale(roi_gray)    for (ex,ey,ew,eh) in eyes:        cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(255,255,0),2)        cv2.imshow('img',img)        cv2.waitKey(0)    cv2.destroyAllWindows()
I have created the output of the above program as a video and embedded it below. If you see the program is not able to properly detect the eye of the person wearing glasses. Probably we need a more sophisticated algorithm for it.
Let’s implement one more use case from the haar cascade classifier. In this use-case we will be detecting the vehicles from a streaming video.I have implemented these use-cases to show how it works. There are whole bunch of other xmls also for this classifier which you could use to implement to implement few other computer vision cases as well. Here is the github link for the xmls.
The implementation here is same as the one we did for face detection, so I won’t be going in detail explaining the whole process. However there are couple of changes that are there in the code.
In order to detect the features of a vehicle we need to import the haarcascade_car.xml.
Use the VideoCapture of cv2 and store the value in cap
Reading (cap.read()) from a VideoCapture returns a tuple (ret, frame). With the first item you check whether the reading was successful, and if it was then you proceed to use the returned frame.
cap = cv2.VideoCapture('/vehicle.mp4')# Loop once video is successfully loadedwhile cap.isOpened():        time.sleep(.05)    # Read first frame    ret, frame = cap.read()
Now that we have the tuple of (ret, frame) , we will convert the BGR channel image to gray channel. Reasons being the same, we are converting the image to gray scale and using the classifier function detectMultiScale to extract the x-coordinate, y-coordinate, width (w) and height(h), and gray scale is used for better performance throughput.
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)       # Pass frame to our car classifier    cars = car_classifier.detectMultiScale(gray, 1.1, 2)
Based on the extracted features/dimensions of the cars, we will loop through them and draw a rectangle around each frame of the image.
for (x,y,w,h) in cars:        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 255), 2)
Below is the consolidated code for that:
import timeimport numpy as npimport cv2# Create our body classifiercar_classifier = cv2.CascadeClassifier('\haarcascade_car.xml')# Initiate video capture for video filecap = cv2.VideoCapture('/vehicle.mp4')# Loop once video is successfully loadedwhile cap.isOpened():        time.sleep(.05)    # Read first frame    ret, frame = cap.read()    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)       # Pass frame to our car classifier    cars = car_classifier.detectMultiScale(gray, 1.1, 2)        # Extract bounding boxes for any bodies identified    for (x,y,w,h) in cars:        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 255), 2)        cv2.imshow('Cars', frame)if cv2.waitKey(1) == 13: #13 is the Enter Key        breakcap.release()cv2.destroyAllWindows()
Now that we have the complete code, let’s check the output for it.
The implementation is completely the same as the vehicle detection. The only difference here will be, we will be using the haarcascade_fullbody.xml to identify the features of the pedestrian’s body.
Below is the code for it:
import numpy as npimport cv2# Create our body classifierbody_classifier = cv2.CascadeClassifier('\haarcascade_fullbody.xml')# Initiate video capture for video filecap = cv2.VideoCapture('/moskva.mov')# Loop once video is successfully loadedwhile cap.isOpened():        # Read first frame    ret, frame = cap.read()gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)    # Pass frame to our body classifier    bodies = body_classifier.detectMultiScale(gray, 1.1, 3)        # Extract bounding boxes for any bodies identified    for (x,y,w,h) in bodies:        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 255), 2)        cv2.imshow('Pedestrians', frame)if cv2.waitKey(1) == 13: #13 is the Enter Key        breakcap.release()cv2.destroyAllWindows()
Let’s check how the output looks like in the below video:
Although haarcascade classifier is pretty helpful, there are few drawbacks of this approach.
The most challenging part of this is accurately specifying the parameter value of scaleFactor and minNeighbors of the detectMultiScale function. It is pretty common to run into scenarios where we need to tune both the parameters on a image-by-image basis which is a big turn off when it comes to an image detection use-case.The scaleFactor is basically used to control the image pyramid which in turn is used to detect the object at various scales of an image. If the scaleFactor is too large then chances are that the image detection will not be accurate and we will be missing objects at scales that fall in between the pyramid layers.However, if we decrease the value of scaleFactor then you will get many layers of pyramids on the same image scale which makes detection slower and increases false-positives.
The most challenging part of this is accurately specifying the parameter value of scaleFactor and minNeighbors of the detectMultiScale function. It is pretty common to run into scenarios where we need to tune both the parameters on a image-by-image basis which is a big turn off when it comes to an image detection use-case.
The scaleFactor is basically used to control the image pyramid which in turn is used to detect the object at various scales of an image. If the scaleFactor is too large then chances are that the image detection will not be accurate and we will be missing objects at scales that fall in between the pyramid layers.
However, if we decrease the value of scaleFactor then you will get many layers of pyramids on the same image scale which makes detection slower and increases false-positives.
Stay tuned for the upcoming posts where we will see other cool approaches to detect images in a more efficient way using some sophisticated deep learning algorithms.
Also, if you want to learn about devops, please refer to this website.