Detecting Pikachu on Android using Tensorflow Object Detection | by Juan De Dios Santos | Towards Data Science

Deep inside the many functionalities and tools of TensorFlow, lies a component named TensorFlow Object Detection API. The purpose of this library, as the name says, is to train a neural network capable of recognizing objects in a frame, for example, an image.
The use cases and possibilities of this library are almost limitless. It could be trained to detect people in an image, cats, cars, raccoons and many more. Due to this reason, I became interested in trying it myself with a custom model trained on my own. What is my use case? Detection of Pikachu.
The purpose of this article is to describe the steps I followed to train my own custom object detection model — and to showcase my Pikachu detection skills — so that you can try it on your own. First, I will start with an introduction of the package by summarizing some of the details explained in the original paper. Secondly, I will continue with how I converted my Pikachu images into the right format and created the dataset. Afterwards, I will write, as detailed as possible, about the training process, and how to evaluate it, as well as the final model, using Tensorflow’s web UI, TensorBoard. Lastly, I will demonstrate how to use the model in a Python notebook, and the process of exporting it to Android.
The code I used for this project is available at my Github (juandes/pikachu-detection). Every script mentioned in this document should be available there.
Before I start, since I am sure most of you are curious, this is an example of the Pikachu detection.
This package is TensorFlow’s response to the object detection problem — that is, the process of detecting real-world objects (or Pikachus) in a frame. According to the documentation and the paper that introduces the library , what makes it unique is that it is able to trade accuracy for speed and memory usage (also vice-versa) so you can adapt the model to suit your needs and your platform of choice, such as a phone. The library includes many out-of-the-boxes object detection architectures such as SSD (Single Shot Detector), Faster R-CNN (Faster Region-based Convolutional Neural Network), and R-FCN (Region-based Fully Convolutional Networks), as well as several feature extractors like MobileNet, Inception, Resnet; these extractors are really significant because they play a huge part in the speed/performance trade-off of the system.
Moreover, the library also provides several already trained models ready to be used for detection, the option to train in Google Cloud, plus the support of TensorBoard to monitor the training.
Now that we know a bit about the system used for this experiment, I will proceed to explain how you can build your own custom model.
Before we begin, please make sure you have TensorFlow installed on your computer. Otherwise, see the instructions here on how to install it. Next, clone the repo where the Object Detection API is included. This is the link: https://github.com/tensorflow/models.
Once you have cloned the repo, navigate to the “research” directory and execute:
# From tensorflow/models/research/protoc object_detection/protos/*.proto --python_out=.
This compiles the Protobuf libraries. If you do not have Protobuf installed, you can download it from here: https://developers.google.com/protocol-buffers/docs/downloads
Lastly, you need to add the libraries to the PYTHONPATH. This can be done by executing:
# From tensorflow/models/research/export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
For more details about all the dependencies needed, please see the official documentation, available here: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md
Now let’s see how I went from having many Pikachu images to a nice and tidy dataset readable by TensorFlow.
Creating the dataset is the first of the many steps required to successfully train the model, and in this section, I will go through all the steps needed to accomplish this.
For this project, I downloaded 230 medium-sized images of Pikachu into a directory named images. For better results, I tried to obtain different angles and shapes of Pikachu, but let’s be honest, Pikachu is a yellow long-eared mouse that does not exist, so it was a bit difficult to find a huge variety of images of it.
Once you have acquired all the images, the next step is labelling them. What does this mean? Since we are doing object detection, we need a ground truth of what exactly the object is. For this, we need to draw a bounding box around the object, to let the system know that this “thing” inside the box is the actual object we want to learn. The software I used for this task is a Mac app called RectLabel.
This is how an image with the bounding box looks like:
In RectLabel, you need to set a label to each bounding box of your image, in my case, the label is “Pikachu”. Once you have labelled all the images, you will notice that you have one directory named “annotations” with many XML files describing the bounding box of each image.
Once all the images were labelled, my next step was to split the dataset into a train and test dataset. Inside the same directory where the images are, I created a directory named “train” and “test” and added around 70% of the images and their respective XML to the training directory and the remaining 30% to the test directory.
Following the separation of the dataset, the only thing missing is to convert our images and their XML into a format that is readable by TensorFlow. Said format is called “tfrecord”, and to generate it from our images, two steps are needed. Firstly, for the sake of simplicity, the data from both set of XML (training and testing), were converted into two CSV files (again, training and testing) using a modified version of the xml_to_csv.py code provided on datitran’s GitHub (https://github.com/datitran/raccoon_dataset). Then, using the CSV files, the tfrecord dataset is created using the script generate_tfrecord.py (also from datitran’s). Keep in mind that before running the script you have to specify the class of your objects in the function class_text_to_int.
*EDIT (23.01.17): Will Brickner reported that the xml_to_csv.py is not working as intended with the latest version of RectLabel, and he wrote a new script that is more robust. It is available here: https://gist.github.com/wbrickner/efedf8ab0ce1705de1372c1e2f49dd98 Many thanks :)
A “labels” map indicating the labels and their indexes is required. This is how they look.
item {  id: 1  name: 'Pikachu'}
Replace the “Pikachu” label with yours, and important, always start with index 1 because 0 is reserved. I saved this file as object-detection.pbtxtin a new directory named “training”.
The complete training process is handled by a config file known as the “pipeline”. Said pipeline is divided into five main structures that are responsible for defining the model, the training and evaluation process parameters, and both the training and evaluation dataset inputs. The skeleton of the pipeline looks like this:
model {(... Add model config here...)}train_config : {(... Add train_config here...)}train_input_reader: {(... Add train_input configuration here...)}eval_config: {}eval_input_reader: {(... Add eval_input configuration here...)}
But! You do not have to write the whole pipeline from scratch. In fact, the TensorFlow devs recommend that the training should use one of their own and already trained models as a starting point. The reason behind this is that training a completely new model from scratch might take a considerable amount of time. Thus, TensorFlow provides several config files that only need a minimal amount of changes to make it work in a new training environment. The model I used was ssd_mobilenet_v1_coco_11_06_2017 found here.
For my training I used the config file ssd_mobilenet_v1_pets.config as a starting point. The changes I did were to the variable num_classes since my number of classes is just one, num_steps to halt the training earlier, fine_tune_checkpoint to point to the location of the model downloaded, and the input_path and label_map_path variables of the train_input_reader and eval_input_reader to point to the training and test dataset and the labels map.
I will not go through every single detail of the config file because honestly, I do not understand some parts of it and because it is too large. However, I would like to explain what SSD and MobileNet means so you can have a bit of knowledge of what is going on under the hood.
SSD, which stands for Single Shot Detector, is a neural network architecture, based on a single feed-forward convolutional neural network. It is called “single shot” because it predicts the class of the image and the position of the box that represents the detection (known as the anchor) during the same step. The opposite of this is an architecture that requires a second component known as the “proposal generator” to predict the exact position of the box.
MobileNet is a convolutional feature extractor, designed to work on mobile devices, that is used to obtain the high-level features of the images.
Once you have the pipeline ready, add it to the “training” directory. Then, proceed to start the training using the following command:
python object_detection/train.py --logtostderr --train_dir=path/to/training/ --pipeline_config_path=path/to/training/ssd_mobilenet_v1_pets.config
This library provides the code needed to evaluate the model during the training and after it. Each time the training produces a new checkpoint, the evaluation tool will perform predictions using the images available in a given directory (in my case I used the images from the test set).
To run the evaluation tool, execute this:
python object_detection/eval.py --logtostderr --train_dir=path/to/training/ --pipeline_config_path=path/to/training/ssd_mobilenet_v1_pets.config --checkpoint_dir=path/to/training/ --eval_dir=path/to/training/
The results of the training and the evaluation phases can be seen using TensorFlow’s visualization platform, TensorBoard. Here, we can monitor several metrics such as the training time, total loss, number of steps and many more. Something really cool is that TensorBoard also works while the model is being trained, making this a great tool to make sure the training is heading in the right direction.
To execute TensorBoard, run the following command:
tensorboard --logdir=path/to/training/
Once the training is finished, the next step is to export the model so it can be used. The library provides the script, named export_inference_graph.py, needed to perform this step.
Before exporting it, make you sure you have the following files in the training directory:
model.ckpt-${CHECKPOINT_NUMBER}.data-00000-of-00001,model.ckpt-${CHECKPOINT_NUMBER}.indexmodel.ckpt-${CHECKPOINT_NUMBER}.meta
You may have several files with the same format, but with different checkpoint numbers. Thus, just select the desired checkpoint, and execute the following command:
python object_detection/export_inference_graph.py                   --input_type image_tensor                                           --pipeline_config_path=path/to/training/ssd_mobilenet_v1_pets.config --trained_checkpoint_prefix=path/to/training/model.ckpt-xxxxx       --output_directory path/to/output/directory
The output will be a file that holds a “frozen” version of the model named frozen_inference_graph.pb.
At the end of the training phase, the precision of the model was 87%, with a total loss of 0.67. However, during the training, the model reached a maximum precision of 95%. Nonetheless, the model with the highest precision was not performing as good as I expected. For example, it classified many yellow objects (even some people) as Pikachu. On the other hand, I noticed that the model with a precision of 87% was producing less false positives, at the expense of missing some Pikachu. The images below are the total loss and precision graph produced by TensorBoard.
TensorBoard also automatically evaluates some of the images of the evaluation set. What is really nice is that by using a slider, you could see how the confidence of the prediction changes based on which checkpoint of the model is being used.
The image detection package includes a notebook to test one of the pre-trained models TensorFlow provides. Yet, this notebook can be changed to work with the frozen version (the one we exported) of a custom trained model, and so I did. Below you can see some of the detections done in the notebook.
So far, we have trained, exported the model, and evaluated it. Now it is time to import it to Android so we can use it to detect Pikachu using the phone’s camera. Now, this part was somehow really complex and it gave me a lot of headaches, so I will try to explain the main steps as detailed as possible. However, I am really sure that some of you who are replicating this experiment will have some issues so I would like to apologize beforehand if my guide does not support you.
Let’s move on to the Android section of TensorFlow. Firstly, you need to download Android Studio. Once you have it, clone the TensorFlow repo (which you might already have), and in Android Studio import a new project using the directory from the TensorFlow repo you just cloned, called “Android”. As an additional note, I also recommend going through the README to get a bit familiarized with the library.
The README recommends keeping the build as its simplest, and it suggests to change the nativeBuildSystem variable of the Gradle build to none, however, I changed it to cmake to build it (none of the other building alternatives like Bazel and Makefile worked for me).
When the build is finished, the next step is to add the frozen model to the assets directory. Then, also in that folder, create a file called “labels”, and in the first line write ??? (remember when I said the first class is reserved?), and in the second line write the label of your object (in my case I wrote Pikachu).
Then, open the file named “DetectionActitivity.java” located in the “java” directory; this is the code used by the app to perform the detections. Look for the variable TF_OD_API_MODEL_FILE and TF_OD_API_LABELS_FILE and in the first one, change its value to the path of the frozen model located in the assets folder, and in the second one write the path of the file with the labels. Another useful variable you should know of is MINIMUM_CONFIDENCE_TF_OD_API which is the minimum confidence needed it to track a detection. Try to play around with it!
Now we are ready! Click on the run button, select your Android device, and wait a couple of seconds until the app is installed on the phone. Important detail, not one, but four apps, will be installed on the phone, however, the one we that contains the detection model is TF Detect. If everything went fine, and the app launched, find some pictures of your object and see if the model is able to detect them.
And that’s it! Happy detection :)
Here are some of the detections I did in my phone.
EDIT (05.06.2018): A continuation of this work, detecting Pikachu in videos, is available here:
towardsdatascience.com
In this article, I explained all the necessary steps to train a custom model using TensorFlow Object Detection library. At the beginning, I gave a bit of background information about the library and how it works, followed by a guide on how to label and process and images to generate the dataset. Afterwards, I focused on how to do the training. In this section, I talked about the training pipeline, how to evaluate the model using TensorBoard. Then, once the training was done, I went through the procedure of exporting the model and importing in a Python notebook and Android.
The hardest part of this project was getting the model to run on Android. Without exaggerating, this took me about a month because I was getting tons of error, (I even opened an issue on Github) and I almost give up. What solved the issue for me, was to modify, in the export.py file (called by export_inference_graph.py), the parameter optimize_graph to False. Apparently, and as far as I understood, by optimizing the model, my model was creating some nodes not recognizable by the Android library, and thus, it was crashing. I am not saying that this is a solution, but it might be something you could try in case it is not working for you. My recommendation is to not give up and keep trying, so you can impress your friends with your own custom detection system.
Thanks for reading, and I really hope this guide helped some of you. If you have any questions, comments, doubt, or just want to chat, leave a comment and I will be happy to help.