Serving models using Tensorflow Serving and Docker
By: Vimarsh Karbhari
Tags: Deep Learning, Artificial Intelligence, Machine Learning, TensorFlow, Interview

Serving models using Tensorflow Serving and Docker

How to serve machine learning models using Tensorflow Serving and Docker

The tremendous potential with Machine learning can only be realized when models are put in production and users can interact with them directly or indirectly.

What is model serving?

Model serving is the process of exposing of a trained model so that it can be accessed by an endpoint. Endpoint here can mean a direct user or other software.

Prerequisites

Some prerequisites for this process are:

Python (3.5>) and Tensorflow (2.0 >) installed on your local system

Docker is installed on your local system

Model serving process

Installing Tensorflow Serving

Since you have Docker properly installed, you can use it to download TF Serving.

In your terminal run the following command below:

docker pull tensorflow/serving

This takes some time, and when done, will download the Tensorflow Serving image from Docker Hub.

If you are running Docker on an instance with GPU, you can install the GPU version as well:

docker pull tensorflow/serving:latest-gpu

Building, training, and saving a model

For the purpose of this article, we are not going to optimize our models here. We will serve a basic model utilizing a popular image classification model for this demonstration.

The MNIST handwritten digit classification dataset is a very popular image classification task. It contains handwritten digits from humans and the task is to classify these digits into a number between 0 and 9. Because the dataset is very popular, Tensorflow comes prepackaged with it, as such, you can easily load it.

Next we will walk you through loading the dataset, and then build a simple deep learning classifier.

Step 1: Create a new project directory and open it in your code editor. I called mine tf-serving, and I have it open in the IDE.

Step 2: The logic is pretty straight forward, first, you import the necessary packages you’ll be using. In addition, you load the MNIST dataset prepackaged with Tensorflow. Then, you reshape the data to use a single channel (black & white) and then normalize by dividing with 1/255.0.

Finally, you create a simple Convolutional Neural Network (CNN) with 9 classes at the output because you’re predicting for 10 classes (0–9). Then you compile the model by specifying an optimizer, a loss function, and a metric.

For this step in the project folder, create a new script called model.py, and paste the code below:

import matplotlib.pyplot as plt

import time

from numpy import asarray

from numpy import unique

from numpy import argmax

from tensorflow.keras.datasets.mnist import load_data

from tensorflow.keras import Sequential

from tensorflow.keras.layers import Dense

from tensorflow.keras.layers import Conv2D

from tensorflow.keras.layers import MaxPool2D

from tensorflow.keras.layers import Flatten

from tensorflow.keras.layers import Dropout #load MNIST dataset

(x_train, y_train), (x_test, y_test) = load_data()

print(f'Train: X={x_train.shape}, y={y_train.shape}')

print(f'Test: X={x_test.shape}, y={y_test.shape}') # reshape data to have a single channel

x_train = x_train.reshape((x_train.shape[0], x_train.shape[1], x_train.shape[2], 1))

x_test = x_test.reshape((x_test.shape[0], x_test.shape[1], x_test.shape[2], 1)) # normalize pixel values

x_train = x_train.astype('float32') / 255.0

x_test = x_test.astype('float32') / 255.0 # set input image shape

input_shape = x_train.shape[1:] # set number of classes

n_classes = len(unique(y_train)) # define model

model = Sequential()

model.add(Conv2D(64, (3,3), activation='relu', input_shape=input_shape))

model.add(MaxPool2D((2, 2)))

model.add(Conv2D(32, (3,3), activation='relu'))

model.add(MaxPool2D((2, 2)))

model.add(Flatten())

model.add(Dense(50, activation='relu'))

model.add(Dropout(0.5))

model.add(Dense(n_classes, activation='softmax')) # define loss and optimizer

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) # fit the model

model.fit(x_train, y_train, epochs=10, batch_size=128, verbose=1) # evaluate the model

loss, acc = model.evaluate(x_test, y_test, verbose=0)

print('Accuracy: %.3f' % acc) #save model

ts = int(time.time())

file_path = f"./img_classifier/{ts}/"

model.save(filepath=file_path, save_format='tf')

Here in our code, you fit the model for 10 epochs using a batch size of 128. After fitting, you evaluate it on the test data, print the accuracy, and finally save the model.

The code for saving the model is based on timestamp. This is a basic versioning and saving practice and is highly recommended.

The folder structure should be similar to the one shown below:

├── img_classifier

│ ├── 1600788643

│ │ ├── assets

│ │ ├── saved_model.pb

│ │ └── variables

Serving saved model with Tensorflow Serving

Once you have your model saved, and Tensorflow Serving correctly installed with Docker, you are going to serve it as an API Endpoint.

It is worth mentioning that Tensorflow Serving allows two types of API Endpoint — REST and gRPC. In this setup here, we will be using REST since it is easier to use and inspect. It should also be noted that Tensorflow Serving will provision both Endpoints when you run it, so you do not need to worry about extra configuration and setup.

Follow the steps below to serve your model:

First, in your project folder, open a terminal, and add the Docker command below:

docker run -p 8501:8501 --name tfserving_classifier \

--mount type=bind,source=/Users/tf-server/img_classifier/,target=/models/img_classifier \

-e MODEL_NAME=img_classifier -t tensorflow/serving

Command details:

-p 8501:8501: This is the REST Endpoint port. Every prediction request will be made to this port. For instance, you can make a prediction request to http://localhost:8501.

— name tfserving_classifier: This is a name given to the Docker container running TF Serving. It can be used to start and stop the container instance later.

— mount type=bind,source=/Users/tf-server/img_classifier/,target=/models/img_classifier: The mount command simply copies the model from the specified path (/Users/tf-server/img_classifier/) into the Docker container (/models/img_classifier), so that TF Serving has access to it.

-e MODEL_NAME=img_classifier: The name of the model to run. This is the name you used to save your model.

-t tensorflow/serving: The TF Serving Docker container to run.

Let us test by using a new script in your project folder called prediction.py, with the sample code below.

import matplotlib.pyplot as plt

import requests

import base64

import json

import numpy as np

from tensorflow.keras.datasets.mnist import load_data



#load MNIST dataset

(_, _), (x_test, y_test) = load_data()

# reshape data to have a single channel

x_test = x_test.reshape((x_test.shape[0], x_test.shape[1], x_test.shape[2], 1))

# normalize pixel values

x_test = x_test.astype('float32') / 255.0



#server URL

url = 'http://localhost:8501/v1/models/img_classifier:predict'



def make_prediction(instances):

data = json.dumps({"signature_name": "serving_default", "instances": instances.tolist()})

headers = {"content-type": "application/json"}

json_response = requests.post(url, data=data, headers=headers)

predictions = json.loads(json_response.text)['predictions']

return predictions



predictions = make_prediction(x_test[0:4])



for i, pred in enumerate(predictions):

print(f"True Value: {y_test[i]}, Predicted Value: {np.argmax(pred)}")

A sample output would look like:

//output

True Value: 7, Predicted Value: 7

True Value: 2, Predicted Value: 2

True Value: 1, Predicted Value: 1

True Value: 0, Predicted Value: 0

Sometimes, the default port 8501 may be unavailable or in use by other system processes, you can easily change this to another port when running the Docker image.

Once training is done and you have saved the model, TF Serving automatically detects this new model, unloads the old one, and loads the newer version.

Conclusion

Hopefully if you have followed up to this point to sum it up: