Image Classification on Tensorflow Serving with gRPC or REST Call for Inference | by Sushrut Ashtikar | Towards Data Science

Tensorflow provides a variety of ways to deploy the model. I have explored most of the ways of serving a model in production. I was looking out for a simple, secure, and robust solution that should be easy to access on both edge devices as well as servers written on other programming languages. Model Server was perfect for my needs. I connected edge devices through REST API and made internal RPC calls to my other server. I am also able to serve different versions to different devices which I can easily control from a central server. In this article, I have demonstrated how you can make calls to the model server in both ways and find out which method suits your problem statement.
We are going to deploy an image classifier trained on Keras. To make our model run on the model server, we need to export our weights in a specific format as mentioned below.
model_name/    version/        saved_model.pb        variables/           variables.index           variables.data-000-of-n
In TensorFlow 2 we can directly export trained model with tf.keras.models.save_model function.
MODEL_DIR = './cat_dog_classifier'version = 1export_path = os.path.join(MODEL_DIR, str(version))tf.keras.models.save_model(    model,    export_path,    overwrite=True,    include_optimizer=True,    save_format=None,    signatures=None,    options=None)
We will be using saved_model_cli utility for examining graphs. This will be useful for finding out the input and output node name.
saved_model_cli show --dir ./cat_dog_classifier/1/ --all
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list && \curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -sudo apt-get updatesudo apt-get install tensorflow-model-server
We are going to run the model server on default ports 8500 for grpc and 8501 for HTTP. You can also specify different ports if you want.
Assuming your terminal location is in the same folder where the model folder is present.
export MODEL_DIR=$(pwd)/cat_dog_classifiertensorflow_model_server --port=8500 --rest_api_port=8501 --model_name=cat_dog_classifier --model_base_path="${MODEL_DIR}"
After successfully running a model server with your saved model you will get similar output for your model.
2020–06–30 15:22:57.309568: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: cat_dog_classifier version: 1}2020–06–30 15:22:57.311296: I tensorflow_serving/model_servers/server.cc:355] Running gRPC ModelServer at 0.0.0.0:8500 ...[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...2020–06–30 15:22:57.315925: I tensorflow_serving/model_servers/server.cc:375] Exporting HTTP/REST API at:localhost:8501 ...
Necessary Imports:
import grpcimport numpy as npimport nsvision as nvimport tensorflow as tffrom tensorflow_serving.apis import predict_pb2from tensorflow_serving.apis import prediction_service_pb2_grpc
Setting gRPC channel:
label = ['cat', 'dog']GRPC_MAX_RECEIVE_MESSAGE_LENGTH = 4096 * 4096 * 3channel = grpc.insecure_channel('localhost:8500', options=[('grpc.max_receive_message_length', GRPC_MAX_RECEIVE_MESSAGE_LENGTH)])stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)grpc_request = predict_pb2.PredictRequest()grpc_request.model_spec.name = 'cat_dog_classifier'grpc_request.model_spec.signature_name = 'serving_default'
Predicting output from an image:
Make sure you specify correct inputs and outputs to the gRPC request object. In case if you want to know what is the input-output key of your model you can check using metadata which can be found by calling a systematic URL in the browser. In this case, it is:
http://localhost:8501/v1/models/cat_dog_classifier/metadata
My input layer name was conv2d_input and output layer name was dense_1 but it will differ as per model.
image = nv.imread('golden-retriever-royalty-free-image-506756303-1560962726.jpg',resize=(150,150),normalize=True)image = nv.expand_dims(image,axis=0)grpc_request.inputs['conv2d_input'].CopyFrom(tf.make_tensor_proto(image, shape=image.shape))result = stub.Predict(grpc_request,10)result = int(result.outputs['dense_1'].float_val[0])print(label[result])#This printed 'dog' on my console
Rest API requires fewer libraries and less setup than grpc.
Imports:
import jsonimport requestsimport nsvision as nv
Creating JSON for REST call:
label = ['cat','dog']image = nv.imread('cat.2033.jpg',resize=(150,150),normalize=True)image = nv.expand_dims(image,axis=0)data = json.dumps({     "instances": image.tolist()})headers = {"content-type": "application/json"}
Predicting output from an image:
REST API call requires a URL for POST call, the model server has specified a URL format for API calls: http://host_ip:port/version_number/models/model_name:predict
In this case, the API URL will be:
http://localhost:8501/v1/models/cat_dog_classifier:predict
response = requests.post('http://localhost:8501/v1/models/cat_dog_classifier:predict', data=data, headers=headers)result = int(response.json()['predictions'][0][0])print(label[result])#This printed 'cat' on my console
Since it’s a REST API you can connect endpoint to web client like javascript or on the edge devices like Android, Ios, or IoT devices like Raspberry Pi or Jetson Nano.