Tensorflow Serving with Docker. How to deploy ML models to production. | by Vijay Gupta | Towards Data Science

This article will guide you through how you can build and train a simple CNN model and later use this trained model to be served as an endpoint using Tensorflow Serving. Whether you are an Amateur, ML Engineer, Data Scientist or the DevOps guys, I am sure you will be able to deploy the ML model by the end of this article.
Tensorflow Serving is an API designed by Google for production machine learning systems, Google and many big tech companies use this extensively. It makes it easy to deploy your model with the same server architecture and APIs. It works best with a TensorFlow model but I guess it can be extended to serve other kinds of models as well.
The above diagrams show a brief image of how all this process looks like from building the model to serving that model to an endpoint using Tensorflow Serving. The best option to serve most kinds of models might be to run a centralized model on a server where any kind of device maybe it’s desktop, mobile, or embedded devices can request from. Then the server would do the inference for you and return the predictions. From that prediction, you can render it to any device. A big advantage in this type of architecture is that suppose you have several clients accessing your endpoint which is centralized on a server. Everyone gets to access the same model version and can enjoy the updated version seamlessly also scaling for large number requests becomes easy by just adding load balancers. It would have become difficult if you would have deployed the model on each client machine, hence managing versions and putting out new updates becomes challenging.
Now let us get into the WORK MODE.
We will be training a neural network model to classify images on clothing (i.e mnist fashion dataset), saving trained TensorFlow based model and serve it with Tensorflow Serving. Although, more focus will be on serving the model part.
You will find all the codes in the below Git Repository. Star this repository and follow me on Medium, I will be posting more updates in the coming weeks on Tensorflow algorithms, ML Model pipelines and examples to integrate with cloud services like AWS and Azure.
It will be easy if you run these codes in Google Collab.
github.com
import sys# Confirm that we're using Python 3assert sys.version_info.major is 3, 'Oops, not running Python 3.'import tensorflow as tffrom tensorflow import kerasimport numpy as npimport osimport subprocess
Pre-Processing Data before feeding it to the model.
# Model VersionVERSION = '00000123'fashion_mnist = keras.datasets.fashion_mnist(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()# scale the values to 0.0 to 1.0train_images = train_images / 255.0test_images = test_images / 255.0# reshape for feeding into the modeltrain_images = train_images.reshape(train_images.shape[0], 28, 28, 1)test_images = test_images.reshape(test_images.shape[0], 28, 28, 1)print('\ntrain_images.shape: {}, of {}'.format(train_images.shape, train_images.dtype))print('test_images.shape: {}, of {}'.format(test_images.shape, test_images.dtype))
Building and Training the CNN Model on Fashion MNIST Data
model = keras.Sequential([keras.layers.Conv2D(input_shape=(28,28,1), filters=8, kernel_size=3,strides=2, activation='relu', name='Conv1'),keras.layers.Flatten(),keras.layers.Dense(10, activation=tf.nn.softmax, name='Softmax')])model.summary()epochs = 15model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy'])model.fit(train_images, train_labels, epochs=epochs)test_loss, test_acc = model.evaluate(test_images, test_labels)print('\nTest accuracy: {}'.format(test_acc))
Saving the trained model in TensorFlow protobuf format.
saved_model_path = model.save(f"./tmp/{VERSION}", save_format='tf')
SavedModel CLI helps you to examine your model and inspect the model’s SignatureDef's. It gives you details about the input Tensor dtype and shape the model was trained on. So, you have to give input to the model in the same format and get the output as defined by SignatureDef's.
import subprocesssubprocess.run([f"saved_model_cli show --dir ./tmp/{VERSION}/ --all"],shell=True)# Zipping the model as model.tar.gzsubprocess.run([f"tar cvfz model.tar.gz tmp/{VERSION}/"],shell=True)
We will be serving the model on our local machine, so we need the compressed model file and the JSON file which will be input to the model.Download the compressed model file, so that you can deploy it on your local machine( assuming you have ran the whole code in Google Collab).
from google.colab import filesfiles.download('model.tar.gz')
Making a JSON file which will be containing the inputSignatureDef's required for the model, this will then be passed to the model endpoint.
import json# Wrap bitstring in JSONdata = json.dumps({"signature_name": "serving_default", "instances": test_images[0:3].tolist()})json_file = open('predict.json', 'w')json_file.write(data)json_file.close()files.download('predict.json')
Install Docker from their official site.
Quick links to download:
Docker for macOS
Docker for Windows 10 Pro or later
Let us start with pulling the latest Tensorflow Serving image
docker pull tensorflow/serving
Running the Serving image with our model deployed on the REST API endpoint.
docker run -p 8501:8501 --mount type=bind,source=/path/to/the/unzipped/model/tmp/,target=/models/fashion_mnist -e MODEL_NAME=fashion_mnist -t tensorflow/serving
Now your docker container is running with Tensorflow Serving Model Server, binding the REST API port 8501 and mapped the model from our host to where models are expected in the container. Also an environment variable is passed which is important to query our model.
You can query the model using predict API
curl -d ‘{"signature_name": "serving_default", "instances": [[[[0.0], [0.0]................[0.0]]]]}’ -X POST http://localhost:8501/v1/models/fashion_mnist:predict
Or you can run the below script in your local machine to get the prediction
Awesome, Now your model just won’t get lost in your local system but you can deploy your model to production so that people can make use of it.
You have successfully created an endpoint using Tensorflow Serving in your local system. I will be posting more future blogs to run this same and several other types of model on cloud services.
If this helped you, Tweet about the post and let people deploy their model because this is a bit complex and crucial part of the machine learning lifecycle.
Star this repository and follow me on Medium, I will be posting more updates in the coming weeks on Tensorflow algorithms, ML Model pipelines and examples to integrate with cloud services like AWS and Azure.