Train a Hand Detector using Tensorflow 2 Object Detection API in 2021 | by Aalok Patwardhan | Towards Data Science

I wanted to make a computer vision application that could detect my hands in real-time. There were so many articles online that used the Tensorflow object detection API and Google Colab, but I still struggled a lot to actually get things working. The reason? Versions of libraries and code change!
Here is an example of using this detector in a virtual theremin:
This article should guide you on what works right now (March 2021). I will assume you know basic Python skills, and enough know-how to look up what you don’t know from other tutorials! 😂
Things we will be using:
Google Colab
Tensorflow Object Detection API 2
The Egohands dataset: http://vision.soic.indiana.edu/projects/egohands/
Steps:1. Set up environment2. Download and Install Tensorflow 2 Object Detection API3. Download dataset, generate tf_records4. Download model and edit config5. Train model and export to savedmodel format
Acknowledgements:Big thanks to github users molyswu, datitran, and gilberttanner from whom I have taken some code and modified it slightly. Please do check out their tutorials as well.
Open up a new Google Colab notebook, and mount your Google drive. You don’t need to, but it’s super handy incase you disconnect from your session, or just want to come back to it again.
from google.colab import drivedrive.mount('/content/drive')%cd /content/drive/MyDrive
We are now inside your Google Drive. (You may need to change that last %cd incase your drive mounts to a slightly different path).
Google Colab will be using Tensorflow 2, but just in case, explicitly do this:
%tensorflow_version 2.x
The first thing is to download and install Tensorflow 2 Object Detection API The simplest way is to first go into your root directory and then clone from git:
%cd /content/drive/MyDrive!git clone https://github.com/tensorflow/models.git
Then, compile the protos — there is no output by the way.(protoc should work as it is, because Google Colab already has it installed):
%cd /content/drive/MyDrive/models/research!protoc object_detection/protos/*.proto --python_out=.
Now install the actual API:
!cp object_detection/packages/tf2/setup.py . !python -m pip install .
Test your tensorflow object detection API 2 installation! Everything should be “OK”. (It’s ok if some of the tests are automatically skipped).
#TEST IF YOU WANT!python object_detection/builders/model_builder_tf2_test.py
I’ll describe at a top level what you need to do here, as this part hasn’t really changed much from other tutorials. In summary, we are going to download the Egohands dataset but only use a subset of the many many images there, since we are doing transfer learning.We will split them into a train directory and a test directory, and generate .xml files for each of them (containing the bounding box annotations for each image).
I have created a general script which:
Downloads the whole Egohands dataset and extracts it
Only keeps a small number (4) of folders from it
Splits the images into a train and test set
Creates annotation .csv files with bounding box coordinates
First we need to make sure you are in your root directory and then clone my git repo.
%cd /content/drive/MyDrive!git clone https://github.com/aalpatya/detect_hands.git
From my downloaded repo, copy the egohands_dataset_to_csv.py file into your root location and run it. This will do everything for you — by default it will only take 4 random folders (so 400 images) from the actual Egohands dataset, split them into a train and test set, and generate .csv files.
!cp detect_hands/egohands_dataset_to_csv.py .!python egohands_dataset_to_csv.py
Big shoutout to https://github.com/molyswu/hand_detection, from whom I got the original script. I’ve just tidied it up a bit and made a few tweaks. 😁
The test_labels.csv and train_labels.csv files that were just created contain the bounding box locations for every image, but surprise surprise, Tensorflow needs that information in a different format, a tf_record.
We will create the required files train.record and test.record by using generate_tfrecord.py from my git repo, (I modified this from the wonderful datitran’s tutorial at https://github.com/datitran/raccoon_dataset).
%cd /content/drive/MyDrive!cp detect_hands/generate_tfrecord.py .# For the train dataset!python generate_tfrecord.py --csv_input=images/train/train_labels.csv  --output_path=train.record# For the test dataset!python generate_tfrecord.py --csv_input=images/test/test_labels.csv  --output_path=test.record
/content/drive/MyDrive (or whatever your root is called)  |__ egohands  |__ detect_hands  |__ images    |__ train      |__ <lots of images>      |__ train.csv    |__ test      |__ <lots of images>      |__ test.csv  |__ train.record  |__ test.record
Get the download link of a model you want from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md
Here I am using SSD Mobilenet V2 fpnlite 320x320. I found that some models did not work with tensorflow 2, so use this one if you want to be sure.
%cd /content/drive/MyDrive!wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz# Unzip!tar -xzvf ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz
First, create a file called label_map.pbtxt which contains your hand class. It should look like this:
item {  id: 1  name: 'hand'}
Or, you can just note down the path to the one already in my git repo which you should already have: /content/drive/MyDrive/detect_hands/model_data/ssd_mobilenet_v2_fpn_320/label_map.pbtxt
Next we will edit the pipeline.config that came with the downloaded tensorflow model. It will be inside the model directory of the model you downloaded from the tensorflow model zoo. For example: ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/pipeline.config
Near the start of pipeline.config:
Change the number of classes to 1:
Towards the middle/end of pipeline.config:
Set the path to the model checkpoint. We only want the beginnig part of the checkpoint name until the number. For example: “ckpt-0”, and not “ckpt-0.index”.
Set the checkpoint type as “detection”
You may also want to change the batch size. In general the lower the batch size the quicker your model loss will drop, but it will take longer to settle at a loss value. I chose a batch size of 4 because I just wanted training to happen faster, I’m not looking for state of the art accuracy here. Play around with this number, and read this article.
At the end of pipeline.config:
Set the path to label_map.pbtxt (there are two places to do this, one for testing and one for training)
Set the path to the train.record and test.record files
First we’re going to load up the tensorboard, so that once training begins we can visualise the progress in nice graphs.
The logdir argument is the path to the log directory that your training process will create. In our case this will be called output_training, and the logs automatically get stored in output_training/train.
%load_ext tensorboard%tensorboard --logdir=/content/drive/MyDrive/output_training/train
Now begin training, setting up the right paths to our pipeline config file, as well as the path to the output_training directory (which hasn’t been created yet).
%cd /content/drive/MyDrive/models/research/object_detection/#train !python model_main_tf2.py \--pipeline_config_path=/content/drive/MyDrive/detect_hands/model_data/ssd_mobilenet_v2_fpn_320/pipeline.config \--model_dir=/content/drive/MyDrive/output_training --alsologtostderr
This will begin the process of training, and you just sit back and wait. Either you wait a long time until the training process finishes, or just cancel the process after some time (perhaps you see on the loss graph that the loss is levelling off). It’s ok to do that, because the training process keeps saving model checkpoints.
Now we will export the training output into a savedmodel format so that we can use it for inference.
%cd /content/drive/MyDrive/models/research/object_detection!python exporter_main_v2.py \--trained_checkpoint_dir=/content/drive/MyDrive/output_training \--pipeline_config_path=/content/drive/MyDrive/detect_hands/model_data/ssd_mobilenet_v2_fpn_320/pipeline.config \--output_directory /content/drive/MyDrive/inference
The important bit of this whole thing is the inference folder. It is the only thing we actually need if we want to perform inference.
CONGRATULATIONS! You have trained a hand detector! 🎈🎉🎊
The model can be loaded with tensorflow 2 as
detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)
and from there you can use the detect_fn function and go ahead with inference, but I’ll leave that for another tutorial 😉