Deploying Pretrained TF Object Detection Models on Android | by Shubham Panchal | Towards Data Science

Deploying machine learning models on mobile devices is the new phase of ML that’s about to begin. Vision models, mostly object detection models, have already made their way to mobile devices along with speech recognition, image classification, text completion etc. These models, run usually run on GPU-enabled computers, have tons of use-cases when deployed on mobile devices.
In order to demonstrate an end-to-end example on how to bring a ML model, specifically an object-detection model to Android, we’ll use Victor Dibia’s hand detection models for our demonstration, from the victordibia/handtracking repo. The model can detect human hands from an image and is made using the TensorFlow Object Detection API. We’ll use the trained checkpoints from Victor Dibia’s repo and convert them to the TensorFlow Lite ( TFLite ) format, which can be used to run the model on Android ( or even iOS, Raspberry Pi ).
Next, we move on to the Android app and create all the necessary classes/methods required to get the model running and also display its predictions (the bounding boxes ) over the live camera feed.
Let’s get started!
Conversion of Model Checkpoints to TFLite
👉 1. Setting Up the TF Object Detection API
👉 2. Converting the Checkpoints to Frozen Graphs
👉 3. Converting the Frozen Graphs to the TFLite Buffer
Integrating the TFLite model in Android
👉 1. Adding Dependencies For CameraX, Coroutines and TF Lite
👉 2. Initializing CameraX and ImageAnalysis.Analyzer
👉 3. Implementing the Hand Detection Model
👉 4. Drawing the Bounding Boxes over the Camera Feed
Realtime Depth Estimation In Android ( GitHub )
MLKit Selfie Segmentation in Android ( GitHub )
Age + Gender Estimation in Android with TensorFlow ( GitHub )
Using FaceNet For On-Device Face Recognition With Android ( GitHub )
Text2Summary Android ( GitHub )
Q-Learning With The Frozen Lake Environment In Android ( GitHub )
Hyperparameter Optimization With Genetic Algorithms in Kotlin ( GitHub )
Exploring Random Forests In Light Of Kotlin ( GitHub )
Coding Feed-Forward Neural Networks in Kotlin (or Android) ( GitHub )
Gaussian Naive Bayes ( for Iris Classification ) in Android ( GitHub )
How I made Skinly for Melanoma Detection in Android ( GitHub )
Designing Decision Trees From Scratch on Android ( GitHub )
Text Classification in Android with TensorFlow ( GitHub )
Sarcasm Detection using Word Embeddings in Android ( GitHub )
Hands-on With Multiple Linear Regression on Android
Bayes Text Classification in Kotlin for Android without TensorFlow
Introducing TensorFlow Lite Android Support Library
Our first step will be to convert the trained model checkpoints, provided in Victor Dibia’s repo (MIT License), to the TensorFlow Lite format. TensorFlow Lite provides an efficient gateway to run TensorFlow models on Android, iOS and micro-controller devices. In order to run the conversion scripts, we need to set up the TensorFlow Object Detection API on our machine. You may also use this Colab Notebook to perform all conversions in the cloud.
I recommend you to use the Colab Notebook ( especially for Windows ) as I personally went into a number of errors while doing so.
The TensorFlow Object Detection API provides a number of pretrained object detection models which can be fine-tuned on custom datasets and deployed directly into mobile, web, or the cloud. We’ll only require the conversion scripts that help us convert the model checkpoints into a TF Lite buffer.
The hand detection model was itself made using the TF OD API with TensorFlow 1.x. So, first we need to install TensorFlow 1.x or TF 1.15.0 ( the latest version in the 1.x family ) and then clone the tensorflow/models repo which contains the TF OD API.
Also, we’ll clone Victor Dibia’s handtracking repo to get the model checkpoints,
!git clone https://github.com/victordibia/handtracking
Now in the models/research/object_detection directory, you’ll observe a Python script export_tflite_ssd_graph.py which we’ll use to convert the model checkpoints to a TFLite-compatible graph. The checkpoints could be found in the handtracking/model-checkpoint directory. ssd stands for ‘Single Shot Detector’ which is the architecture for the hand detection model whereas mobilenet denotes the backbone architecture of MobileNet ( v1 or v2 ) which is a specialized CNN architecture for mobile devices.
The exported TFLite graph contains fixed input and output nodes. We can find the names and the shapes of these nodes ( or tensors ) in the export_ssd_tflite_graph.py script. Using the script, we’ll convert the model checkpoints to a TFLite compatible graph, given three arguments,
pipeline_config_path : Path to the .config file which contains the configuration of the SSD Lite model used.trained_checkpoint_prefix : Prefix of the trained model checkpoints we wish to convert.max_detections : The number of bounding boxes to be predicted. This is important as it is an important parameter for the non maximum suppression postprocessing operation added to the graph.
pipeline_config_path : Path to the .config file which contains the configuration of the SSD Lite model used.
trained_checkpoint_prefix : Prefix of the trained model checkpoints we wish to convert.
max_detections : The number of bounding boxes to be predicted. This is important as it is an important parameter for the non maximum suppression postprocessing operation added to the graph.
After the script has been executed, we are left with two files, tflite_graph.pb and tflite_graph.pbtxt which are TFLite compatible graphs.
Now we’ll use a second script ( or more precisely, a utility ) to convert the frozen graphs, generated in step 2, into the TFLite buffers ( .tflite ). As TensorFlow 2.x ruled out the use of Session and Placeholder we can’t convert frozen graphs to TFLite here. This was one of the reasons why we installed TensorFlow 1.x in step 1.
We’ll use the tflite_convert utility to convert the frozen graph to a TFLite buffer. We can also use the tf.lite.TFLiteConverter API, but we’ll stick to the command line utility for now.
Once the execution is done, you’ll see a model.tflite file in the outputs directory. In order to check the input/output shapes, we’ll load the TFLite model using tf.lite.Interpreter and call .get_input_details() or .get_output_details() to get the input and output details respectively.
Tip: Use pprint to get a beautiful output.
Once we’ve got our TFLite model with all the details of its input and output shapes, we’re ready to run it in an Android app. Create a new project in Android Studio or feel free to fork/clone the GitHub repo to get started!
As we’re going to detect hands on a live camera feed, we’ll need add CameraX dependencies to our Android app. Similarly, in order to run TFLite models, we’ll require the tensorflow-lite dependency along with the Kotlin Coroutines dependency that helps us run the model asynchronously. In the app-level build.gradle file, we’ll add the following dependencies,
Make sure you add aaptOptions{ noCompress "tflite" } so that the model isn’t compressed by the system to make your app’s size smaller. Now, in order to place the TFLite model in our app, we’ll create an assets folder under app/src/main . Paste the TFLite file ( .tflite ) in this folder.
We’ll use a PreviewView from the CameraX package to display the live camera feed to the user. Over it, we’ll place an overlay, called the BoundingBoxOverlay , to draw the bounding boxes over the camera feed. I won’t be discussing the implementation here, but you can learn it from the source code or this story of mine,
proandroiddev.com
As we’re going to predict the bounding boxes for hands on live frame data, we’ll also require an ImageAnalysis.Analyzer object which returns every frame from the live camera feed. See this snippet from FrameAnalyzer.kt ,
BitmapUtils contains some useful static methods to manipulate Bitmap s. isFrameProcessing is a Boolean variable which determines whether the incoming frame must be dropped or passed to the model. As you may observe, we’re running the model in a CoroutineScope and hence you’ll observe no lags while the model produces the inference.
Next, we’ll create a class called HandDetectionModel which will handle all the TFLite operations and return the predictions given an image ( as a Bitmap ).
We’ll understand each of the terms separately in the above snippet,
modelImageInputDim is the size of the input image for our model. Our model would take in an image of size 300 * 300.maxDetections represents the maximum number of predictions made by our model. It determines the shapes of boundingBoxesTensorShape , confidenceScoresTensorShape , classesTensorShape and numTensorShape .outputConfidenceThreshold is used to filter the predictions made by our model. This is not NMS but we only take the boxes whose score is greater than this threshold.inputImageProcessorQuantized and inputImageProcessorNonQuantized are instances of TensorOperator which resize the given images to a size of modelImageInputDim * modelInputImageDim . In case of a quantized model, we standardize the given image with mean and standard deviation both equal to 127.5.
modelImageInputDim is the size of the input image for our model. Our model would take in an image of size 300 * 300.
maxDetections represents the maximum number of predictions made by our model. It determines the shapes of boundingBoxesTensorShape , confidenceScoresTensorShape , classesTensorShape and numTensorShape .
outputConfidenceThreshold is used to filter the predictions made by our model. This is not NMS but we only take the boxes whose score is greater than this threshold.
inputImageProcessorQuantized and inputImageProcessorNonQuantized are instances of TensorOperator which resize the given images to a size of modelImageInputDim * modelInputImageDim . In case of a quantized model, we standardize the given image with mean and standard deviation both equal to 127.5.
Now, we’ll implement a method run() which will take a Bitmap image and output the bounding boxes in the form of List<Prediction> . Prediction is a class which holds the predicted data, like the confidence score and bounding box coordinates.
confidenceScores , boundingBoxes , classes and numBoxes are the four tensors which will hold the outputs of our model. The processOutputs method will filter the bounding boxes and return only those boxes whose confidence score is greater than the threshold.
Once we’ve received the bounding boxes, we’ll like to draw them over the camera feed, just as we do with OpenCV. We’ll create a new class BoundingBoxOverlay and add it in the activity_main.xml . The class looks like,
That’s All! We’ve just implemented a hand detector in an Android app! You may run the app after reviewing all the code.
Hope you enjoyed this story! Feel free to express your thoughts at equipintelligence@gmail.com or in the comments below.
Have a nice day ahead, dear developer!