Deep Learning with Tensorflow: Part 4 — face classification and video inputs | by Matteo Kofler | Towards Data Science

Hi everybody, welcome back to my Tenserflow series, this is part 4. This will probably be the last part of the series, since we already learned so much:
Part 1 was all about theory, we looked at the logic and functionality of neural networks and Tensorflow.
We performed image classification with geometric shapes in the second part.
In part 3, we successfully generated our own music and text.
Since image classification is the most interesting part when it comes to neural networks (at least in my opinion), we’ll do it again in this part. But instead of geometric shapes, we’ll classify images of people, before we move on to rebuild our classifier to take in video inputs. Let’s go for it.
You probably won’t remember what the second part was about, so here’s a short summary:
We built a program which loaded the pre-trained Inception v3 model from Google, removed the old top layer and trained a new one on the geometric shapes classes we wanted to add. The re-training worked in two phases — Bottleneck and Training. Bottleneck processed our images to a meaningful and compact summary of them, Training actually trained the top layer of our neural network. If it is still unclear, please read part 2 ;)
Last time, we added four classes of geometric shapes to our inception model. This time, we’ll classify three different people: “Steve Jobs”, “Bill Gates” and “Mark Zuckerberg”. Let’s see if Inception is strong enough to differentiate between those people’s faces:
Start by cloning my repository from GitHub, type:
git clone https://github.com/koflerm/tensorflow-image-classifier.git
Next up, we need to add our training images and label them. Build a structure like this:
/--- /training_dataset|    ||    --- /billgates|    |    billy.jpg|    |    bill_gates.png|    |    ...|    ||    --- /stevejobs|         stevejobs.jpg|         jobs.jpg|         ...|    ||    --- /markzuckerberg|         zuckerberg46.jpg|         zuckerberg2020.jpg|         ...
To download the images, i use a Chrome Extension named ‘Bulk Download Images’. This extension allows you to automatically download Google Images to your computer. Image format isn’t important!
After setting up the data, let the training begin! Execute the train.sh script by double clicking it. This script installs the Inception model and initiates the re-training process for the specified image data sets.
Once the process is completed, our training accuracy should be somewhere around 90 percent.
After re-training the model, it’s now time to test the model with other images. Download and copy them into the root directory and type:
python classify.py downloadedBillGates.jpg
Bill Gates looks straight to the camera, the light isn’t bad, so no problem.
Surprisingly, the right person gets recognized again. The image resolution was poor (300x300), and Zuckerberg isn’t even the center of the image. Well done, Inception!
That’s a tricky one! Inception is trained for single-label image classification, which means that it adds up to a total score of one. Multi-label classification is not possible, and since two of our classes are shown in the image, both can’t get a high score. If i wanted the image to be classified more as “Steve Jobs”, I had to use a more accurate and bigger training set for his class.
Again, the image gets classified as Gates over Jobs, this time by a huge margin. I guess a more accurate training set respectively more training data in that exact lighting conditions could win Steve Jobs that duel.
Finally, the man gets recognized! But that wasn’t really hard for our model, was it (great lighting, he looks straight to the camera, ...)
Gates gets recognized, but so does Mark Zuckerberg (although he isn’t on the picture). I guess Inception thought of Sandro Rosell (on the left) as Mark Zuckerberg, but yeah, definitely a mismatch!
(Optional): If you now want to add new classes, you can combine the training and testing by typing:
python retrain.py — bottleneck_dir=tf_files/bottlenecks — how_many_training_steps=500 — model_dir=inception — summaries_dir=tf_files/training_summaries/basic — output_graph=tf_files/retrained_graph.pb — output_labels=tf_files/retrained_labels.txt — image_dir=training_dataset — eval_step_interval=100 & python classify.py image.jpg
Note: image.jpg is the image you want to test your model with afterwards
There you have it, we can classify faces with Inception. But for now, let’s move on to rebuild our classifier for video inputs.
So far, we downloaded our test data image-by-image, but that’s laborious, isn’t it? It would be far easier to test our model with a video input. We only had to insert a video once, but got the classification done frame-by-frame. Way easier. Therefore, let’s build such a system:
Basically, it’s not much what changes. We only need to touch the classify.py script. I’ll guide you through the changes step-by-step:
pip install opencv-py 
First off, open a terminal and install OpenCV, a library for video processing and image analysis.
import cv2import math
We then need to somehow load and process our video. We do that by importing the OpenCV module. The import of the math package is actually optional, you only need it if you don’t want to process every single frames.
So, that’s the important part of the code. If you want to get into it a little bit deeper, please read the comments, i think they speak for themselves. The principle is simple: When we get a video frame, we store it and immediatly read it again for classification purpose, before we output the scores and the frame itself in a seperate window.
But first, I think you what you have to in the meantime ... Execute train.sh and wait until the process is completed :)
Then, the fun part starts. I’ll test my model on the following video:
Execute python classify.py myVideo.mp4 Next, we have to wait a long time, until Bill Gates finally gets filmed for the first time. What’s the reaction of the classifier?
As we can see, the model actually classifies him correctly, although Bill Gates’ head is cut off a little bit. We sure know that our model does work with videos now, but does it recognize Steve Jobs as well?
Yes. It’s actually even better with Jobs, the score is about 0.9 . But one question still remains open: what happens if both men are in the same frame?
Well, the frame gets classified as “Jobs”. As explained earlier, the scores add up to one, both men can’t get a high score.
It’s time to take on a new challenge now: Hand drawings. Will our model recognize hand drawn objects?We’ll again add our four geometric classes “circle”, “triangle”, “square” and “plus” to do so:
So, at first we have to add our classes to the model again, in case you deleted them since part 2. In the end, there should be a structure like this in the training folder.
/--- /training_dataset|    ||    --- /billgates|    --- /stevejobs|    --- /markzuckerberg|    --- /circle|    --- /square|    --- /plus|    --- /triangle
Fill all these folders with training images. I’ll fill them with images from the Google Search, although these images won’t be hand drawings! It’ll be interesting to see how that goes.
Next up, training. Run the train.sh script to add the new classes to the model, before filming your own custom video of geometric shapes. You can also use mine, find it as math_own_old.mp4 in my repo on GitHub.
So, let’s classify some frames. Type python classify.py math_own_old.mp4 in your CLI and look, if your model is able to recognize the right shapes.
What a great start! Squares are a success, but what about the other objects?
Well, there you have it. Our model does not recognize the hand drawn triangle, it thinks of it as square. Maybe the next one ...
It was worth a try, but it again thinks of it as a square. Strangly, the plus seems to be invisible for the model as well. The circle is our last hope!
Yes, the circle gets classified correctly! But as seen, it is very hard for a model that was trained by computer generated images to recognize badly resolved hand drawn pictures.
So, there you have it, it is possible to classify people with Inception and Tensorflow. Moreover, you learnt how to rewrite some code to use video as your input source. This was the last part of the series, I hope you enjoyed the journey. I for myself definitely learned a lot. If you have any type of feedback for me, i’d apprecitate it ✌