Deploy Keras Models using TensorFlow Serving — TF 2.x | by Pranit Bankar | Towards Data Science

Created a great Deep Learning model? Congratulations!
Don’t know how to create a Deep Learning model using Keras? No worries!
I shall cover how to create a basic and super simple Keras Model and how to deploy it on your local machine. This model would definitely be simpler than whatever you plan to deploy (well, I hope so!). But since the goal of this post is to enable you to deploy your Keras Model, I want to ensure that we don’t deviate from our goal.
Now why do we need to deploy it on our local server? To verify that whatever problem we intend to solve with our model, can be solved in the real world. You would expect that your model works in the same way and gives the same performance on the real world data that it gave on your test input in your local IDE or notebook. But the truth is it will hardly ever do that. If it does, great! But you wouldn’t want to realize the gap between your test data and the real world data after you have deployed your model for consumption.
What if the real world images are different from the white background studio images you used to train, validate and test your model? What if it takes too much time to process your data on the network calls due to latency issues than it did in your IDE? Finding these problems early are key to successfully transitioning your model from your cozy laptop to the reality outside.
TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs. TensorFlow Serving provides out-of-the-box integration with TensorFlow models, but can be easily extended to serve other types of models and data.
If you are used to building your models using TensorFlow or Keras, then the easiest way of deploying your models is by using TensorFlow Serving.
Most of the TensorFlow documentation is written for TensorFlow-1.0 and it sadly doesn’t work as is for TensorFlow-2.0. Hence the need for this blog.
You can find my notebook here: (You can also run it in Google Colab)
[If you already have your model ready in SavedModel format, skip this part.]
Here I shall be creating a model that predicts a linear relationship:
y = 2x + 1
I used Google Colab but you can use any tool of your choice as long as the generated model remains the same.
Let’s dig in.
We first load the necessary libraries
We then create our dataset
Then we build our model. Since it is a simple linear relationship, a single neuron is enough.
We now train our model for 20 epochs. After 20 epochs, our model’s losses were around 2e-12 for training as well as validation. Then, we evaluate our model on the test data. Our test loss came out to be around 2e-12. You can try predicting some values to verify the results like I did here. The output I received is 15.866.
Everything looks great.Let’s now save and download our model in the SavedModel format.
From this step onwards, the process would be independent of the type of input and output data used. It could be numeric arrays, texts, images, audios or videos. In Colab, you should be able to see a folder named ‘linear_model’ created in your directory. The ‘export_path’ variable here indicates that our model is named ‘linear_model’ and this is the first version of it. Having a version number is mandatory for deploying using TensorFlow Serving so ensure that you have your ‘export_path’ in the form of {MODEL}/{VERSION}, where VERSION is numeric without any alphabets or special characters.
Now to download this model, we will zip this folder and then use ‘google.colab.files’ to download the zip file.
Once you extract this zip file on your local machine, you should be able to see a folder named ‘linear_model’, which contains a folder named ‘1’, which contains your variables and model architecture.
This is a one-time setup activity.
To start our local server we need a TensorFlow Serving instance on our local machine. Instead of downloading and installing all the necessary libraries, we shall use the recommended way of using Docker.
You can read more details about it in the Guide here.
We won’t be following all the steps mentioned in the Guide, as some of the things are specific for TF 1.0 and some things are specific to reusing already available models.
Once you download Docker on your system from here, go ahead and finish the installation steps. You would need to restart your system so save all your work.
Once the installation is finished successfully, go to Command Prompt (Mac and Linux users, kindly use appropriate tools) and type
docker pull tensorflow/serving
That’s it! Let’s now proceed to deploy our model.
What if I told you that deploying the model is just one line of command script.
All you need is the absolute path to your ‘linear_model’ folder. Do not forget to use the absolute path as this would cause errors that you would need to spend time and break your head to solve.
My ‘linear_model’ was saved inside ‘D:/my_own_models/’. So my command looked like:
docker run -p 8038:8501 — mount type=bind,source=D:/my_own_models/linear_model,target=/models/linear_model -e MODEL_NAME=linear_model -t tensorflow/serving
This is all a single line. For your subsequent models, you need to just change your ‘source’ path. Changing your ‘target’ and MODEL_NAME is optional, but, of course, necessary based on the context.
Now let us try to understand what the above command script indicates. The generic form of this script is
docker run -p {LOCAL_PORT}:8501 — mount type=bind,source={ABSOLUTE_PATH},target=/models/{MODEL_NAME} -e MODEL_NAME={MODEL_NAME} -t tensorflow/serving
{LOCAL_PORT}: This is the local port of your machine, so make sure you do not have anything else running there. We are mapping it to the 8501 port exposed for REST API calls by TensorFlow Serving
{ABSOLUTE_PATH}: This is the absolute path for your model. This tells the TensorFlow Serving where your model is located at (obviously).
{MODEL_NAME}: This is the service end-point for your REST API call with the prefix ‘/models/’. Do not change the ‘/models/’ prefix in the target variable and only change the {MODEL_NAME} part based on your need
Once you see the below message in your command window, your model has been hosted successfully. You can also see a container running successfully in your Docker dashboard.
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
I used Postman for testing my queries, but you can use any form of API calls.
[Note: You won’t be able to make direct calls from browser or any other host to your REST API since CORS is not enabled by TensorFlow Serving. However, there are ways to achieve a call from the browser to your model that I would cover in a separate post.]
You need a POST query to test our model. Our request URL looks like:
http://localhost:8509/v1/models/linear_model:predict
Again, the generic form is:
http://localhost:{LOCAL_PORT}}/v1/models/{MODEL_NAME}:predict
Add ‘Content-Type’ as ‘application/json’ in your header and your body as:
{     "instances": [[          0     ]]}
Ensure that your JSON key is ‘instances’ and your values are inside an array. Since our input is [0], we write it as [[0]].
Remember, that we created a model to predict y = 2x + 1 which means for input value 0, our prediction should have value 1 (or close to it).
Let’s send our query. The response looks like:
{     "predictions": [[          0.999998748     ]]}
Well, that looks pretty much like 1 to me. Play around with your model by making POST queries.
We were able to create a model from scratch and deploy it on our local server. This is a very good way to see if your model would work the same way in the real world as it does on your IDE.
As I previously mentioned, however, you won’t be able to make calls to this model apart from Postman (or similar tools). How to overcome that? I shall cover that in a separate post as this is already becoming too long.
If you have reached here, thanks for reading. If you have any queries, suggestions or comments, please feel free to comment on this post. This is my first blog on Machine Learning and I would gladly appreciate any and all feedback.