Building a Multiple Object Detection Model with TensorFlow’s Object Detection API
By: Ronak Bhatia
Tags: Machine Learning, Artificial Intelligence, TensorFlow, Multiple Object Detection, Object Detection

This post isn’t meant to be an in-depth explanation of machine or deep learning, but rather, provide a practical guide on setting up object detection for projects. This blog post will cover building a custom object detection system using TensorFlow’s Object Detection API. I have written another blog post on how to build a custom, single object detection model using Fast AI, which is linked here!

Multiple Object Detection on a Web Application running on Chrome

This is part one of two on building a custom object detection system for web-based and local applications. The second part is written by my coworker, Allison Youngdahl, and will illustrate how to implement this custom object detection system in a React web application and on Google Cloud Platform (GCP).

While there are a few examples of how to implement object detection models online, many are deprecated, do not provide clear documentation on troubleshooting, do not provide customization instructions, or do not provide instruction on exporting the machine learning model. For TensorFlow specifically, the Object Detection API is difficult to navigate, and the troubleshooting process took quite a bit of time. It is my hope that this blog post provides some troubleshooting tips and easy, step-by-step instructions for setting up a custom object detection system. Additionally, many existing tutorials or examples use an ML model that is very slow and would not be practical on mobile.

For this blog post, I ran everything with the following specs: macOS Catalina, Version 10.15.4, 16GB RA, 2.3 GHz, 8-Core Intel i9 .

Background

The impetus for this project is for use in an object-detection web application for detecting products in real-time. In the age of coronavirus, this application is useful as it allows customers to immediately gain information about products without requiring people to physically touch them. Additionally, the app is accessible as a web application rather than a smartphone-specific application and creates exciting opportunities for personalization due to its portability, ease of use, and detection capabilities.

References & Acknowledgements

Before beginning the post, I’d like to acknowledge the following people, of which the work is heavily based off of or referenced. Tanner Gilbert has done some great work with documenting how to use the Object Detection API on his YouTube channel. His work and code reference previous work done by Dat Tran and EdjeElectronics on using the Object Detection API. I found the following guide by Adrià Gil to be incredibly useful for troubleshooting and learning how to properly export TensorFlow models. Additionally, I’d like to thank and acknowledge Allison Youngdahl for her help with proofreading this article and for assistance with troubleshooting as well.

Decision Making

When building the web application, the team looked into several tools for object detection. The two tools that came to mind first were TensorFlow and Fast AI. However, the team also looked into MediaPipe as a potential solution. The team eventually chose TensorFlow because of available documentation on porting TensorFlow models to web applications. Fast AI doesn’t, at the time of writing this blog, have an explicit tutorial on multiple object detection — a desired feature of the web application. The team strayed away from MediaPipe due to a lack of available documentation as well. This blog post will walk through TensorFlow’s Object Detection API for multiple object detection, which was used to build a model for the web application.

TensorFlow’s Object Detection API

TensorFlow’s Object Detection API is an open-source framework that’s built on top of TensorFlow to construct, train, and deploy object detection models. There are new models being added even today, with the most recent addition in March 2020 at the time of writing this article. By employing transfer learning (repurposing a pre-trained model for use with items outside the original training data set), the Object Detection API powers multiple object detection for custom items provided you have an appropriately built/sized dataset.

Building a Custom Model with TensorFlow’s Object Detection API

Disclaimer: For the object detection API, I am writing the instructions assuming that you are using a Mac. If you are following along and use Windows, I cannot guarantee that the same steps will work for you. Thank you for your understanding.

Step 1) Clone the Repository and Install Dependencies

The first step is to clone the TensorFlow models repository and set up the Object Detection API. Tanner Gilbert has dockerized this process, which is available here. If doing it manually, you can begin by first cloning the TensorFlow models repository by typing: git clone https://github.com/tensorflow/models . After cloning the repository it is a good idea to install all the dependencies. But first, we should probably install Anaconda. Assuming that you have homebrew installed, you’re going to want to install Anaconda via brew cask install anaconda . Then, you need to insert the following line in your ~/.bash_profile :

export PATH=”/usr/local/anaconda3/bin:$PATH”

This will enable things to work. The terminal command to get this working:

echo ‘export PATH=/usr/local/bin:$PATH’ >>~/.bash_profile

Now, setting up a conda environment. I found a pretty helpful conda cheat sheet online. If you run conda info , you can check if anaconda was installed properly. To create a conda environment with a specific version of Python, run the following code (replace parenthesis as necessary):

# GENERAL

conda create — name (name of project) python=(version you want)

# EXAMPLE

conda create — name tf-object-detection python=3.7.4

Now, you need to activate the environment by doing the following: conda activate tf-object-detection .

Now, installing those pesky dependencies. I would suggest using conda to install but you can also use pip, both of which are shown below:

pip install — user Cython

pip install — user contextlib2

pip install — user pillow

pip install — user lxml

pip install — user jupyter

pip install — user matplotlib

OR

conda install Cython

conda install contextlib2

conda install pillow

conda install lxml

conda install jupyter

conda install matplotlib

conda install tensorflow=1

Installing the COCO API

COCO is a large image dataset designed for object detection, segmentation, person keypoints detection, stuff segmentation, and caption generation. If you want to use the dataset and evaluation metrics, you need to clone the cocoapi repository and copy the pycocotools subfolder to the tensorflow/models/research directory. Here’s what that looked like on my local machine:



cd cocoapi/PythonAPI

make

cp -r pycocotools <path_to_tensorflow>/models/research/

cp -r pycocotools /Users/**put username here**/Desktop/deep-learning-multiple-shoe-training-and-porting-model-tensorflow/Tensorflow-Object-Detection-API-Train-Model/models/research git clone https://github.com/cocodataset/cocoapi.git cd cocoapi/PythonAPImakecp -r pycocotools /models/research/cp -r pycocotools /Users/**put username here**/Desktop/deep-learning-multiple-shoe-training-and-porting-model-tensorflow/Tensorflow-Object-Detection-API-Train-Model/models/research

Using make won’t work on Windows. To install the cocoapi on Windows the following command can be used:

pip install “git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI"

Protobuf Installation & Compilation

The Tensorflow Object Detection API uses .proto files. These files need to be compiled into .py files in order for the Object Detection API to work properly. Google provides a program called Protobuf that can compile these files. Protobuf can be downloaded here. Place the downloaded file anywhere you want (for example in the Desktop folder). The specific file I needed to download was the following: protoc-3.11.4-osx-x86_64.zip . After extracting the folder, you need to go into models/research and use protobuf to extract python files from the proto files in the object_detection/protos directory.

The official installation guide uses protobuf like:

./bin/protoc object_detection/protos/*.proto — python_out=.

This script should work if you installed and did everything correctly. The steps below are if, for some reason, they aren’t working (which is mostly an issue if you’re using a Windows computer). Sometimes, the * , which stands for all files, doesn’t work for people so you can use this Python script to execute the command for each .proto file.

import os

import sys

args = sys.argv

directory = args[1]

protoc_path = args[2]

for file in os.listdir(directory):

if file.endswith(“.proto”):

os.system(protoc_path+” “+directory+”/”+file+” — python_out=.”)

This file needs to be saved inside the research folder and I named it use_protobuf.py. I had renamed the downloaded protoc folder from protoc_macosx_version to protoc and moved it to the research folder. Again, the python script command using use_protobuf is only if the original protoc script command isn’t working! Now we can use it by going into the console and typing:

python use_protobuf.py <path to directory> <path to protoc file>

In my case, I had to run the following commands:

xattr -d com.apple.quarantine protoc/bin/protoc

xattr -d com.apple.quarantine protoc/bin/

protoc/bin/protoc object_detection/protos/*.proto — python_out=.

Adding Necessary Environment Variables & Finishing the TensorFlow Object Detection API Installation

Lastly, we need to add the research and research slim folder to our environment variables and run the setup.py file. To add the paths to environment variables in Linux you need to type (in terminal):

export PYTHONPATH=$PYTHONPATH:<PATH_TO_TF>/TensorFlow/models/research export PYTHONPATH=$PYTHONPATH:<PATH_TO_TF>/TensorFlow/models/research/object_detection export PYTHONPATH=$PYTHONPATH:<PATH_TO_TF>/TensorFlow/models/research/slim

In my case, I would run the following:

export PYTHONPATH=$PYTHONPATH:/Users/**insert username here**/Desktop/deep-learning-multiple-shoe-training-and-porting-model-tensorflow/Tensorflow-Object-Detection-API-Train-Model/models/research export PYTHONPATH=$PYTHONPATH:/Users/**insert username here**/Desktop/deep-learning-multiple-shoe-training-and-porting-model-tensorflow/Tensorflow-Object-Detection-API-Train-Model/models/research/object_detection export PYTHONPATH=$PYTHONPATH:/Users/**insert username here**/Desktop/deep-learning-multiple-shoe-training-and-porting-model-tensorflow/Tensorflow-Object-Detection-API-Train-Model/models/research/slim

To run the setup.py file we need to navigate to ../models/research and run:

# From within /models/research/

python setup.py build

python setup.py install

Now, run the object_detection_tutorial.ipynb from the object_detection folder (Tanner Gilbert has created this helpful Jupyter Notebook, which is available here). You can also check everything is working by simply importing object_detection inside a python shell: import object_detection . If there’s no output, it’s likely working. If things go well, your jupyter notebook looks like the following:

Gathering Data for Transfer Learning

Now that the Tensorflow Object Detection API is ready to go, we need to gather the images needed for training. To train a robust model, we need lots of pictures (at least 50 for each item being trained with 50 images of various items in the same photo) that should vary as much as possible from each other. That means that they should have different lighting conditions, different backgrounds, and lots of random objects in them. You can either take the pictures yourself or you can download them from the internet. I’ve included a separate repository that walks through formatting images and exporting them here. The only difference is you should also run conda install pyqt .

Inside the Fix_Image folder, there is a folder called images which should be empty. Before training the model or creating the testing or training directories, it’s essential to reformat the images (at least, for the way I’m doing it in TensorFlow) to reduce the resolution of the images. This is crucial to prevent the training process from taking too long. Let’s say that you have taken your photos and you’ve added them to the images folder. The next step is to make sure you’re in the Fix_Image directory. Next, run the following command in terminal:

python transform_image_resolution.py -d images/ -s 800 600

This automatically changes the resolution of all the photos in the images folder via a Python script. The script has run properly if you get no output. You can check if the script has actually worked by going into the images folder and seeing the resized images — they will look different than the original photos!

After you have all the images move about 80% to the object_detection/images/train directory and the other 20% to the object_detection/images/test directory. Make sure that the images in both directories have a good variety of classes.

It’s important to note that having .png and .jpg copies with the same name may mess up things when generating XML files as noted by folks who have followed my instructions. For example, if you have picture1.jpg and picture1.png, this may create issues when generating XML files (next step). One way to prevent yourself from having duplicate photo names is the following method (which I haven’t tested):

Download the following script here. Place the downloaded files in a folder called google-images-download. Then navigate to the folder where the python script is and execute (where the item of interest is the object you’re interested in collecting images for):

python google_images_download.py — keywords “(item of interest)” — limit 100 — format jpg

Labeling Data

With all the pictures gathered, we come to the next step — labeling the data. Labeling is the process of drawing bounding boxes around the desired objects. LabelImg is a great tool for creating an object detection dataset.

LabelImg supports two formats, PascalVOC and Yolo. For this tutorial make sure to select PascalVOC. LabelImg saves a xml file containing the label data for each image. These files will be used to create a tfrecord file, which can be used to train the model. The code and documentation for this part is available through my Github in the following repository.

At the end of things, it should look something like the following for the training/test folders (I’m just showing a snippet of the test folder).