Deploy Tensorflow Object Detection API on Kubernetes with Python, Flask and Docker | by Neeraj Krishna | Towards Data Science

The TensorFlow Object Detection API is an open source framework built on top of TensorFlow that makes it easy to construct, train and deploy object detection models.This article demonstrates how you can serve the Tensorflow Object Detection API with Flask, Dockerize the application and deploy it on Kubernetes using the Google Kubernetes Engine.
Once deployed, you can incorporate the API into your custom application, be it a mobile application or a web application to suit your needs.
These are the steps we will be doing:
Set up the Tensorflow Object Detection API
Serve the API using Flask
Containerize the application using Docker
Deploy the containerized application on Kubernetes
Note: We will not be covering the Tensorflow Object Detection API in-depth since that is not the main emphasis of this article. You can visit the official repo if you are interested. It’ll be really helpful if you’re acquainted with it. We also will not go through the entire process of setting up the environment since we can cut it down using Docker. However, you can refer to this amazing article if you are interested.
Clone my repository into the workspace. We will be working with this code throughout the article.
git clone https://github.com/wingedrasengan927/Tensorflow-2-Object-Detection-API-Flask-Application.git
Install Docker in your system if you haven’t already. A Docker container basically packages up code and all its dependencies so the application runs quickly and reliably in any environment. It includes everything needed to run an application and isolates the dependencies from the main environment.
Inside the repo you have cloned, you will find a Dockerfile. Dockerfiles describe the different processes that are assembled to build an Image and can also contain some metadata describing how to run a container based on this image. Our Dockerfile looks like this:
To build an image from the Dockerfile run the following from inside the repo:
docker build -t tf-od-api:latest .
-t indicates the tag you give to the Docker image. The building takes a lot of time. If you see something like this:
Successfully built 64edf7b4c21fSuccessfully tagged tf-od-api:latest
It means the image was successfully built. Now let’s run a container from the image. Run the following command:
docker run -d -p 5000:5000 tf-od-api:latest
This ensures that we detach from the running container and the container is exposed on the port 5000.
Now let’s test our application. This can be done by running the script client.py which is inside the repo, with the necessary arguments (you need to have opencv and numpy installed in your workspace to run the script):
python client.py --address http://127.0.0.1:5000 --image_path images/test_image.jpg --output_dir outputs/
If you see something like this as the output, then our application is working as expected:
Detections are: [{‘person’: ‘79%’}, {‘chair’: ‘34%’}]the image size is: size=1920x1281the image is being written in outputs/
When we build the image from the Dockerfile, we set the environment necessary to run the Tensorflow Object Detection API, this includes cloning the repository, compiling the protocol buffers, setting the environment variables, downloading the model and when we are ready, we clone the repository which contains the code for serving the application with Flask, and we launch the application on port 5000 and we make sure to expose that port when we run the container.
This is the application code:
You can change the parameters, pre-trained model in this file. Make sure you the changes are reflected in the Dockerfile, you can use COPY in Dockerfile to copy the changed repo from your local environment into the Dockerfile instead of cloning it.
On the client side, the request we send to the endpoint contains the image data. We can use API testing tools like Postman or the curl command to send data. The response we get back contains the detections in the image, the image size and image data with the bounding box overlay. We write this image to output_dir
You can run the application on your local system if you have set up the environment which includes adding the required paths to the PATH environment variable.If the Docker container did not run as expected, you can check the logs by running docker logs container_id to check what went wrongThis is a base version of the API and you can customize it according to your needs.
You can run the application on your local system if you have set up the environment which includes adding the required paths to the PATH environment variable.
If the Docker container did not run as expected, you can check the logs by running docker logs container_id to check what went wrong
This is a base version of the API and you can customize it according to your needs.
Before proceeding, you need to make sure that you have a working account on Google Cloud Platform, have the cloud SDK installed, and have created a project.
Right now we have the Image in our local environment, to deploy we need to host it on the cloud. In this section we will push our Docker Image to Google Container Registry .
First we need to configure Docker to use gcloud as a credential helper. To do so, run the command:
gcloud auth configure-docker
To push any local image to the Container Registry, we need to first tag it with the registry name and then push the image. To tag, run the following command:
docker tag tf-od-api gcr.io/${PROJECT_ID}/tf-od-api
gcr.io is the Hostname which specifies the location or repository where you will store the image.
To push the tagged image, run:
docker push gcr.io/${PROJECT_ID}/tf-od-api
To check if the image has been pushed, list out the images. To do so, run:
gcloud container images list
You should see the image name in the output if it was successfully pushed.
NAMEgcr.io/my-project-21212/tf-od-api
You definitely may have heard of Kubernetes, It has become the standard for deploying and managing containerized applications. In this section we will deploy our application on Kubernetes using the Google Kubernetes Engine.
Kubernetes is a container orchestration system. It helps deploy and manage containerized applications on a cluster. A Kubernetes cluster consists of a Master, which is responsible for managing the cluster and Nodes which are worker machines that run applications.
First, let’s install kubectl. kubectl is used to communicate with Kubernetes. To install kubectl, run:
gcloud components install kubectl
Let’s create a two-node cluster named object-detection. As mentioned, a Node is a worker machine in Kubernetes which may be a virtual machine or a physical machine, depending on the cluster. To create the cluster run the following command:
gcloud container clusters create object-detection --num-nodes=2OUTPUTNAME              LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION    NUM_NODES  STATUSobject-detection  us-central1-a  1.14.10-gke.27  35.193.127.147  n1-standard-1  1.14.10-gke.27  2          RUNNING
After the Cluster is created, It spins up two VM instances on Google Compute Engine which are Cluster’s Node Workers. To see them, run the following command:
gcloud compute instances listOUTPUTNAME                                             ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP   STATUSgke-object-detection-default-pool-cd7fa3ff-3zd6  us-central1-a  n1-standard-1               10.128.0.4   34.67.58.20   RUNNINGgke-object-detection-default-pool-cd7fa3ff-z2hw  us-central1-a  n1-standard-1               10.128.0.5   35.222.93.20  RUNNING
Now that we have a cluster up and running, let’s deploy our containerized application. To do so, we have to create a Kubernetes Deployment. The Deployment instructs Kubernetes how to create and update instances of our application. When we create a Deployment, we have to specify the container image for our application and the name of the deployment. Run the following command to create a deployment:
kubectl create deployment object-detection --image=gcr.io/${PROJECT_ID}/tf-od-api
When we create a deployment, Kubernetes creates Pods to host the application instance. A Pod represents a group of one or more application containers. A Pod runs on a Node.To see the Pods created by Deployment, run:
kubectl get podsOUTPUTNAME                                READY   STATUS    RESTARTS   AGEobject-detection-7c997cbfd8-5nz4j   1/1     Running   0          2m11s
We have to allow traffic from the internet to the application so that we can access it. To do so, run:
kubectl expose deployment object-detection --type=LoadBalancer --port 80 --target-port 5000
The port flag specifies the port number configured on the Load Balancer, and the target-port flag specifies the port number that the tf-od-api container is listening on.
To get the external IP, run the following command:
kubectl get serviceOUTPUTNAME               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGEkubernetes         ClusterIP      10.51.240.1    <none>          443/TCP        30mobject-detection   LoadBalancer   10.51.245.79   35.193.230.83   80:31667/TCP   4m57s
Now let’s check if our application is accessible. To do so we’ll run the same client.py but we have to give the external IP as the address argument:
python client.py --address http://35.193.230.83/(my external IP) --image_path images/input_image.jpg --output_dir outputs/OUTPUTDetections are: [{‘person’: ‘79%’}, {‘chair’: ‘34%’}]the image size is: size=1920x1281the image is being written in outputs/
If you see an output which is similar to when we ran it before, Congratulations! You’ve successfully deployed a containerized Tensorflow Object Detection API application on Kubernetes.
Do not forget to remove the cluster once you are done as it may incur charges. To do so, run the following command to delete the service:
kubectl delete service object-detection
And to delete the Cluster and it’s resources, run:
gcloud container clusters delete object-detectionOUTPUTDeleting cluster object-detection...done.Deleted [https://container.googleapis.com/v1/projects/temp-kubernetes/zones/us-central1-a/clusters/object-detection].
In this article, we have learned how to serve the Tensorflow Object Detection API using Flask, containerize the application using Docker and deploy the application on Kubernetes. Hope it has been of help.
If you have any feedback or want to get in touch in general, please drop me a note on ms.neerajkrishna@gmail.com