Gesture Recognition for Beginners with CNN | by That Data Bloke | Towards Data Science

The CNN or convolutional neural networks are the most commonly used algorithms for image classification problems. An image classifier takes a photograph or video as an input and classifies it into one of the possible categories that it was trained to identify. They have applications in various fields like driver less cars, defense, healthcare etc. There are many algorithms for image classification and in this experiment, we are going to look at one of the most popular algorithms in this genre, called SqueezeNet by DeepScale.
Our goal is to design an application that will use a webcam(or an external camera) as an input device and it will then identify and classify the gesture into one of the categories that we are going to define. In this article, we will see how we can use the SqueezeNet algorithm to design an application that takes different gestures and then triggers certain actions. In short, we can send some commands to our computer (without touching that keyboard or the mouse). Let’s put on our magician’s hat! 🧙‍♂️
The experiment consists of four stages listed below:
Defining our classification categoriesCollect training imagesTrain the modelTest our model
Defining our classification categories
Collect training images
Train the model
Test our model
In our final program, we will be performing some actions. Here is a sample list of actions that I have decided to do. Depending on your specific requirement this number of items in this list would be different. Next, we will map these actions with a category name for ease, to be used later in our program.
Increase the speaker volume (category=up)Decrease the speaker volume (category=down)Mute/Unmute (category=mute)Play/Pause (category=play)Open Google Chrome (category=chrome)
Increase the speaker volume (category=up)
Decrease the speaker volume (category=down)
Mute/Unmute (category=mute)
Play/Pause (category=play)
Open Google Chrome (category=chrome)
Apart from these five categories, we should also have another category (category=nothing) which will be used in our final program when the user shows a hand gesture that our model does not recognize or when there is no input gesture from the user.
Now that we have decided on our six categories, let us see how my gestures look like.
I have created a collage of all the gestures and their respective category name (in red) below.
The next set is to prepare our training images for each of these categories. To gather our training data set (images), we will use our webcam. To make things easy, I have made sure to use images with plain uncluttered background. The following python program will use the OpenCV library to perform this action and store these images into a folder by the name — “training_images”. The program takes two inputs parameters:
python get_training_images.py 50 up
a) Number of images to capture (eg:- 50)
b) Label (or category) name (eg:- up, down, play, chrome etc)
The program stops after capturing that many images. The second parameter instructs the name of the category that these images belong to. The program creates a sub-folder by the category name inside our ‘training_images’ folder and stores all the images over there.
Once you run the program, the webcam feed will turn On. Place your hand within the boundary of the white box and press ‘s’ key to start taking picture of your hand. Try moving your hand around a bit during this process to add some variations in the training data set. I have started here by running the program twice for each category, 50 images with right hand and next 50 images with my left hand. Once the program ends, you should be able to see the images inside the sub-folder by the name of your gesture. Once this process is complete, you will end up with 100 images in each of the folders. You might want to choose a number that suits you, the more the better.
We are going to be using SqueezeNet for this demo. SqueezeNet is a popular pretrained model for image classification problems and it is extremely lightweight with an impressive level of accuracy. We will use Keras library for the training process. Keras is a simple and powerful python library that is widely used in deep learning. It has made it really easy to train neural network models.
Let us look at some of the parameters used for our training.
Sequential: We will use a sequential model, meaning that the layers are arranged in a linear stack (sequence). The layers in a model are added as arguments to this constructor.
Dropout Rate: Neural networks trained on smaller datasets often tend to over-fit and are therefore less likely to be accurate on new data. Theoretically, the best way to train a model could be to try out as many different combination of different parameter values and then take an average of those individual results to come up with a generalized result. But this would require a lot of time and computational resources in training the model multiple times with multiple combinations of these parameters. To get around this, the dropout rate was introduced. Here, some units/nodes in a layer(from input layers or hidden layers, but not from the output layer) are ‘dropped’ which makes that node’s incoming and outgoing connections disappear. Simply put, when this is done multiple times during training, different number of node(s) are dropped from the layer making the layer appear differently in terms of number of nodes and its connections to former layer (roughly simulating multiple models with different layer configurations). A good value for Dropout rate for a layer is between 0.5 – 0.8. In my case, I have found 0.5 to be giving me the best result after trying out few different values. Do note that dropout is used to avoid over fitting. If the model’s accuracy is low, this parameter could be avoided.
Nodes: Number of neurons/nodes in our output layer is equal to the number of classes we are trying to predict.
Input Shape: The input shape that SqueezeNet requires is at least 224 X 224 (with 3 channels for RGB). In this program, we have used 225 X 225.
Choosing an optimizer & loss: Here I have chosen the Adam optimizer. And The loss function is chosen based on the type of problem. For eg:- for a binary classification problem, [loss=’binary_crossentropy’] is better suited and for a multi-class classification problem, [loss=’categorical_crossentropy’] is chosen. Whereas for a regression problem, [loss=’mse’] could be chosen. Since ours is a multi-class classification problem, we will use categorical_crossentropy.
Number of Epochs: The number of epochs is the number of times the entire dataset is passed through the neural network during training. There is no ideal number for this and it depends on the data. In this case, I started with 10, I have used 15. Higher number indicates longer training time.
Activation Function: We are using the ReLU activation function, which is the most common activation function that is used in neural networks due to its computational simplicity (among other advantages). This function returns a zero for negative inputs and the value itself for positive inputs. For most of the modern neural networks, ReLU is the default activation function. Other activation functions that are also used are Sigmoid, Tanh etc.
Pooling: In CNN, it is a common practice to add a pooling layer after the convolution & activation layer. The input image is down sampled or converted to a low resolution version in order to keep only the significant details and remove the finer less important details.
Softmax: The softmax layer is used in multi-class classification right before the output layer. It gives the likelihood of the input image belonging to a particular category.
import numpy as npfrom keras_squeezenet import SqueezeNetfrom keras.optimizers import Adamfrom keras.utils import np_utilsfrom keras.layers import Activation, Dropout, Convolution2D, GlobalAveragePooling2Dfrom keras.models import Sequentialimport tensorflow as tfGESTURE_CATEGORIES=6base_model = Sequential()base_model.add(SqueezeNet(input_shape=(225, 225, 3), include_top=False))base_model.add(Dropout(0.5))base_model.add(Convolution2D(GESTURE_CATEGORIES, (1, 1), padding='valid'))base_model.add(Activation('relu'))base_model.add(GlobalAveragePooling2D())base_model.add(Activation('softmax'))base_model.compile(    optimizer=Adam(lr=0.0001),    loss='categorical_crossentropy',    metrics=['accuracy'])
After training we will store the trained model parameters into a file(gesture-model05_20.h5) which we will be using later during model testing. The entire code is present in the github repository.
model.save("gesture-model05_20.h5")
To test the model, I am using some images of hand gestures captured again with my webcam. The program loads the model file “gesture-model05_20.h5” and it takes the input image as a parameter and predicts the class it belongs to. Before passing the image, we need to ensure that we are using the same dimensions that we used during the training phase.
from keras.models import load_modelimport cv2import numpy as npimg = cv2.imread(<input image file path>)img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)img = cv2.resize(img, (225, 225))model = load_model("gesture-base-rmsprop-model05_20.h5")prediction = model.predict(np.array([img]))
For the following test images the model predicted correctly. However, if the image contains too many other items in the background it isn’t always accurate.
In closing, although this model may not work perfectly every single time and there is a lot we can do to improve its performance, it gives us a starting point and build on these fundamentals from here. Thank your for taking out time to read this article and I hope it helps you to create your first CNN project. I look forward to see your version of this project. Do share with me what you come up with :)
In my next article, we will take a look at how to take actions based on the identified class using Python.
This article uses Python OpenCV. If you are unfamiliar with OpenCV library in Python, here’s a link to my introductory article on this subject.
towardsdatascience.com
medium.com
References & Further Reading:
[1] SqueezeNet, https://codelabs.developers.google.com/codelabs/keras-flowers-squeezenet/
[2] Pooling, 2019, https://machinelearningmastery.com/pooling-layers-for-convolutional-neural-networks/
[3] SqueezeNet Paper, 2016, https://arxiv.org/abs/1602.07360
[4] Alternative OpenCV approach, https://www.youtube.com/watch?v=-_9WFzgI7ak
[5] Dropout: A Simple Way to Prevent Neural Networks from Overfitting, 2014 , http://jmlr.org/papers/v15/srivastava14a.html