How to Build a Smile Detector. Detect Happiness in Python (Tutorial) | by Rohan Gupta | Towards Data Science

Businesses strive to deliver the most important product of all: happiness.
Why? Happiness might just be more than a chemical reaction. A happy customer is more likely to walk through the door again, and data on happiness can help businesses understand which products would do better and have a higher retention rate. Machines can learn to recognize happiness and in this tutorial, I’m going to show you how to create a facial recognition model that can do so.
Let’s start with the basics. How do we express happiness? On the face, it’s mostly through our eyes and smiles. We naturally infer that someone is happy when we see a smile on their face. A smile can have many variations, but its shape is mostly similar to a flatter kind of ‘U’ shaped figure.
:)
In the next steps, I’m going to put down my code and explain what I did to build the happiness detector. To understand the intuition behind the algorithm, check out this previous article on The Viola-Jones Algorithm.
What you need to build happiness detector:
Anaconda Navigator: https://docs.anaconda.com/anaconda/navigator/
OpenCV: https://opencv.org/
Haar Cascades (Link Below)
I’m using Spyder on Anaconda, but you could also use Jupyter Nb I suppose. Once you have everything, get on the IDE (code editor).
You should have a screen like this if you’re using spyder. Before you start coding, make sure to download the haar cascades (next section)
As I had explained in the article cited above, the Viola-Jones algorithm uses haar-like features to detect facial properties. The cascade is a series of filters that will apply one after the other to detect a face through its features.
These filters are stored in their own XML files in the Haar Cascade GitHub Repository.
To build our happiness detector, we need these 3 XML Files: - haarcascade_eye.xml- haarcascade_smile.xml- haarcascade_frontalface_default.xml
These are the cascades for the face, eyes, and smiles. We must have each one of these features if our image is of a happy face. Obtain the code from each of these links, place it into a text editor and save your files according to the names mentioned above. Put all three XML files into the same folder where you will start a python notebook. Now that we have our Haar Cascades, let’s head over to the virtual environment.
— — — — — — — — -Now, its finally time to start coding. — — — — — — — —
import cv2cascade_face = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')cascade_eye = cv2.CascadeClassifier('haarcascade_eye.xml') cascade_smile = cv2.CascadeClassifier('haarcascade_smile.xml')
The only library you need to import is OpenCV. Even though it is a very powerful object-recognition tool, it is not the most powerful one. There are newer better ones out there, but OpenCV still delivers a lot of value and is a good way to understand the basics of object recognition. After importing OpenCV (cv2), I called each of the cascades that were downloaded. For this, I just need to use the OpenCV function called CascadeClassifier.
def detection(grayscale, img):    face = cascade_face.detectMultiScale(grayscale, 1.3, 5)     for (x_face, y_face, w_face, h_face) in face:        cv2.rectangle(img, (x_face, y_face), (x_face+w_face, y_face+h_face), (255, 130, 0), 2) 
Before you define the function for detection, you should know that it is going to be applied to single images, which are then put together to get the end result.
Since the Viola-Jones algorithm works with grayscale images, I input the first argument of this function as grayscale. However, I also want the final output to have the original image in color, so I input another argument img for the original image. Next, I will need to get the coordinates of the rectangles that will detect the face. To define these coordinates, I took 4 tuples: x, y, w, and h. x & y are coordinates of the upper left corner, while w & h are width and height of rectangles respectively. I stored these tuples in a variable face and then used another one of OpenCV's functions called detectMultiScale to actually get these coordinates. So we use our object Cascade_face and apply the detectMultiScale method to it with 3 arguments:- grayscale as the image is being analyzed in b&w- a scale factor of 1.3 (size of the image will be reduced 1.3x)- a minimum number of accepted neighbor zones: 5 neighbors.
Next, to actually draw the rectangles I create a ‘for loop’ with our 4 tuples x_face, y_face, h_face, w_face in faces. In the for loop, I used the rectangle function, which is another OpenCV function. This actually draws the rectangle on your face and is given the following arguments: - ‘img’ because we want rectangle drawn on our original colored image. - coordinates of the upper left corner: x and y- coordinates of the lower right corner: w and h- color of rectangle: I chose a blueish color.- The thickness of edges of rectangles: I picked 2. (doesn’t really matter but 2 is a good choice)
Note: The code below is in continuation (I’ve put it all together at the bottom)
        ri_grayscale = grayscale[y_face:y_face+h_face, x_face:x_face     + w_face]         ri_color = img[y_face:y_face+h_face, x_face:x_face+w_face]
Now that I’m done with the face, I’m going to detect the eyes. This is the only part that’s slightly tricky and a bit harder to understand. Basically, the lines above mention that you are looking for eyes within the face, and so the face becomes our “region of interest”. Since the algorithm works with grayscale images, we input the argument for grayscale. However, I also want to get the colored image, so I will add another argument for the colored image. Essentially, there are 2 regions of interests: one for the grayscale image and one for the original colored image. Subsequently, I create ri_grayscale on the grayscale image with the range of coordinates of the rectangles y:y+h and x:x+w. Then, on the colored image, I create ri_color with the same coordinates for the rectangles.
        eye = cascade_eye.detectMultiScale(ri_grayscale, 1.2, 18)                for (x_eye, y_eye, w_eye, h_eye) in eye:            cv2.rectangle(ri_color,(x_eye, y_eye),(x_eye+w_eye, y_eye+h_eye), (0, 180, 60), 2) 
For the eyes, I repeated the first step with the faces and create an object called eye with cascase_eye and use the detectMultiScale method to find the 4 tuples. I changed the scale factor to 1.2 and the minimum neighbors to 18. ( I had to experiment with these values to get perfect results).
Next, I created a for loop for the eyes as well. You can copy-paste the previous one, just need to change the tuple names and choose a different color.
smile = cascade_smile.detectMultiScale(ri_grayscale, 1.7, 20)         for (x_smile, y_smile, w_smile, h_smile) in smile:            cv2.rectangle(ri_color,(x_smile, y_smile),(x_smile+w_smile, y_smile+h_smile), (255, 0, 130), 2)            return img
Then again for the smile, I repeat the same steps that I did for the face and eyes. When using the detectMultiScale method, I used a scale factor of 1.7 and minimum neighbors of 20. (As I said before, this requires some experimentation to figure out).Once all that is done you just return the original frame.
vc = cv2.VideoCapture(0)while True:    _, img = vc.read()    grayscale = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)     final = detection(grayscale, img)     cv2.imshow('Video', final)    if cv2.waitKey(1) & 0xFF == ord('q'):         break vc.release() cv2.destroyAllWindows() 
I started by creating an object called vc and used the VideoCapture class from OpenCV. It takes only one argument: 0 or 1.0 if you use an internal webcam, 1 if it is external.
Since the detection function works for a single image, I now have to create some sort of loop that allows it to run over a series of images. So I start an infinite while loop which I will break later by using a break function. Now the next lines may seem confusing, but basically, the read method from VideoCapture gets two elements: one of which is the last frame coming from the webcam. Since you only want that, use _, img because the read method returns 2 frames and you only want this one. Now I need to call this method from the VideoCapture class.
I use the cvtColor function that will convert the colored img to grayscale since it needs a b&w frame for the detect function. I call it grayscale and take the cvtColor class, which takes 2 arguments: - the frame (img)- cv2.COLOR_BGR2GRAY — does an average of Blue-green-red to get right shades of grey.Now I created a new variable called ‘final’ and this will be the result of the detect function. Within ‘final’, I use the detection function with arguments grayscale and img.
The imshow function is another OpenCV function that allows us to view the original material coming from the webcam with the rectangle animations. It displays the processed images in an animated way.
Then I used another function to close the window when I’m done using it. I applied an if condition which will terminate the application every time I press the ‘q’ button on the keyboard. This will break the while loop to end the process. Finally, I used the release method to turn off the webcam and the DestroyAllWindows function to terminate the windows.
I’ve pasted my complete code put together below just in case there’s any confusion from the excerpts above.
At first, there were many reasons why the model didn’t work. It would detect a non-smiling face as a smile too. I had to work with the scale factor, as well as the minimum neighbors to get a perfect result.