Deploy TensorFlow models | Towards Data Science

Learn how to deploy your model to production
Updated to tensorflow 1.7
Okay, you have a model and you want to make it accessible from the web. There are several ways you can do that, but the faster and the most robust is TensorFlow serving.
How it works? Basically TensorFlow serving creates a gRPC server where you can send and get a response from your model. You don’t really know the magic behind it to be a wizard.
These are the steps we are going to do:
Make a stupid model as an example, train and store it
Fetch the variables you need from your stored model
Build the tensor info from them
Create the model signature
Create and save a model builder
Download a Docker image with TensorFlow serving already compile on it
Run the docker image with the correct configuration
Create a client in order to send the gRPC request
Make an API
Test it
Code can be found here.
We just need a model to play with. Below there is a very basic model that just take as input and target two random arrays.
import tensorflow as tfimport numpy as npimport os, sysDATA_SIZE = 100SAVE_PATH = './save'EPOCHS = 1000LEARNING_RATE = 0.01MODEL_NAME = 'test'if not os.path.exists(SAVE_PATH):    os.mkdir(SAVE_PATH)data = (np.random.rand(DATA_SIZE, 2), np.random.rand(DATA_SIZE, 1))test = (np.random.rand(DATA_SIZE // 8, 2), np.random.rand(DATA_SIZE // 8, 1))tf.reset_default_graph()x = tf.placeholder(tf.float32, shape=[None, 2], name='inputs')y = tf.placeholder(tf.float32, shape=[None, 1], name='targets')net = tf.layers.dense(x, 16, activation=tf.nn.relu)net = tf.layers.dense(net, 16, activation=tf.nn.relu)pred = tf.layers.dense(net, 1, activation=tf.nn.sigmoid, name='prediction')loss = tf.reduce_mean(tf.squared_difference(y, pred), name='loss')train_step = tf.train.AdamOptimizer(LEARNING_RATE).minimize(loss)checkpoint = tf.train.latest_checkpoint(SAVE_PATH)should_train = checkpoint == Nonewith tf.Session() as sess:    sess.run(tf.global_variables_initializer())    if should_train:        print("Training")        saver = tf.train.Saver()        for epoch in range(EPOCHS):            _, curr_loss = sess.run([train_step, loss], feed_dict={x: data[0], y: data[1]})            print('EPOCH = {}, LOSS = {:0.4f}'.format(epoch, curr_loss))        path = saver.save(sess, SAVE_PATH + '/' + MODEL_NAME + '.ckpt')        print("saved at {}".format(path))    else:        print("Restoring")        graph = tf.get_default_graph()        saver = tf.train.import_meta_graph(checkpoint + '.meta')        saver.restore(sess, checkpoint)        loss = graph.get_tensor_by_name('loss:0')        test_loss = sess.run(loss, feed_dict={'inputs:0': test[0], 'targets:0': test[1]})        print(sess.run(pred, feed_dict={'inputs:0': np.random.rand(10,2)}))        print("TEST LOSS = {:0.4f}".format(test_loss))
Remember: If you use tf.layers.dense the actual output is called with the name that you provide + the function name (lol)
x = tf.placeholder(tf.float32, shape=[None, 2], name='inputs')...pred = tf.layers.dense(net, 1, activation=tf.nn.sigmoid, name='prediction') 
If you run the code it will create the ./save folder if it does not exist or it will just evaluate the model on the test set if there is already one stored instance.
# training the model...EPOCH = 997, LOSS = 0.0493EPOCH = 998, LOSS = 0.0495EPOCH = 999, LOSS = 0.0497saved at ./save/test.ckpt
If we run the script again we see
RestoringTEST LOSS = 0.0710
Perfect! Now we have our saved model
Let’s create a file called serve.py that will handle the logic behind serving our TF model. The first thing to do is to get the variables we need from the stored graph.
import tensorflow as tfimport osSAVE_PATH = './save'MODEL_NAME = 'test'VERSION = 1SERVE_PATH = './serve/{}/{}'.format(MODEL_NAME, VERSION)checkpoint = tf.train.latest_checkpoint(SAVE_PATH)tf.reset_default_graph()with tf.Session() as sess:    # import the saved graph    saver = tf.train.import_meta_graph(checkpoint + '.meta')    # get the graph for this session    graph = tf.get_default_graph()    sess.run(tf.global_variables_initializer())    # get the tensors that we need    inputs = graph.get_tensor_by_name('inputs:0')    predictions = graph.get_tensor_by_name('prediction/Sigmoid:0')
Now we have our variables inputs and predictions . We need to build the tensor info from them that will be used to create the signature definition that will be passed to the SavedModelBuilder instance. Very easily
# create tensors infomodel_input = tf.saved_model.utils.build_tensor_info(inputs)model_output = tf.saved_model.utils.build_tensor_info(predictions)
Now we can finally create the model signature that identifies what the serving is going to expect from the client. In our case
# build signature definitionsignature_definition = tf.saved_model.signature_def_utils.build_signature_def(    inputs={'inputs': model_input},    outputs={'outputs': model_output},    method_name= tf.saved_model.signature_constants.PREDICT_METHOD_NAME)
Here we are calling a utils function from TF that takes the inputs and outputs and the method_name . The last one is just a constant that says what is going to do, in our case we want to PREDICT, they are defined as:
Create a SavedModelBuilder instance and pass all the information we need.
builder = tf.saved_model.builder.SavedModelBuilder(SERVE_PATH)builder.add_meta_graph_and_variables(    sess, [tf.saved_model.tag_constants.SERVING],    signature_def_map={        tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:            signature_definition    })# Save the model so we can serve it with a model server :)builder.save()
In add_meta_graph_and_variables we pass the current session, an array of tags that I have no idea of why we need them, the SERVING constant and the signature_def_map. Again, for the last one, I have no idea of why we need it but basically we are passing a dictionary where we tell that for the DEFAULT_SERVING_SIGNATURE_DEF_KEY we have that signature definition ( the one defined before in “Create the model signature” section).
The full script is here.
Run the script and it creates the serve dir:
First of all, install docker.
Don’t follow the TensorFlow docs since they explain how to setup a docker image and compile TF serving that takes forever.
Now the doc is broken, check here:
https://github.com/tensorflow/tensorflow/issues/19006
Some guy made a docker image with everything already compile on it, so we are going to use that one. The image link:
https://hub.docker.com/r/epigramai/model-server/
Run this command with the correct params
docker run -it -p <PORT>:<PORT> — name <CONTAINER_NAME> -v <ABSOLUTE_PATH_TO_SERVE_DIR>:<SERVE_DIR> epigramai/model-server:light-universal-1.7 — port=<PORT> — model_name=<MODEL_NAME> — model_base_path=<MODEL_DIR>
In our case
docker run -it -p 9000:9000 --name tf-serve -v $(pwd)/serve/:/serve/ epigramai/model-server:light-universal-1.7 --port=9000 --model_name=test --model_base_path=/serve/test
If every works, you should see something like this:
How to do that? Well again, the doc is not really good. So we will use a library that will make things easier.
Run this command to install it using pip , in my case I am using python3
pip3 install git+ssh://git@github.com/epigramai/tfserving-python-predict-client.git
Let’s create a new file called client.py
import numpy as npfrom predict_client.prod_client import ProdClientHOST = '0.0.0.0:9000'# a good idea is to place this global variables in a shared fileMODEL_NAME = 'test'MODEL_VERSION = 1client = ProdClient(HOST, MODEL_NAME, MODEL_VERSION)req_data = [{'in_tensor_name': 'inputs', 'in_tensor_dtype': 'DT_FLOAT', 'data': np.random.rand(1,2)}]prediction = client.predict(req_data, request_timeout=10)print(prediction)
Before run it, be sure you have started the docker container. After lunch the script you should see something like this:
{'outputs': array([[ 0.53334153]])}
A dictionary with outputs as key and the correct prediction as value
It’s time to finally make our model accesible from the web. We will use Flask to create the HTTP server. Let’s refactor our client.py
The client must accept a HTTP call, convert the response data to a numpy array, call the model and return a JSON containing the prediction. Below the final code.
import numpy as npfrom predict_client.prod_client import ProdClientfrom flask import Flaskfrom flask import requestfrom flask import jsonifyHOST = 'localhost:9000'MODEL_NAME = 'test'MODEL_VERSION = 1app = Flask(__name__)client = ProdClient(HOST, MODEL_NAME, MODEL_VERSION)def convert_data(raw_data):    return np.array(raw_data, dtype=np.float32)def get_prediction_from_model(data):    req_data = [{'in_tensor_name': 'inputs', 'in_tensor_dtype': 'DT_FLOAT', 'data': data}]    prediction = client.predict(req_data, request_timeout=10)    return prediction@app.route("/prediction", methods=['POST'])def get_prediction():    req_data = request.get_json()    raw_data = req_data['data']    data = convert_data(raw_data)    prediction = get_prediction_from_model(data)    # ndarray cannot be converted to JSON    return jsonify({ 'predictions': prediction['outputs'].tolist() })if __name__ == '__main__':    app.run(host='localhost',port=3000)
Finally! Me made it! Its time to try it out, open your favore HTTP client, in my case postman, and try to send a JSON with data as key and a vector as value using a POST request to localhost:3000/prediction
After send it we have the response
Perfect! It works.
The final code can be found here
We have seen the best way to serve a TensorFlow model and make it accessible through HTTP request using Flask.
Maybe you can also find worth reading this my other articles:
https://towardsdatascience.com/reinforcement-learning-cheat-sheet-2f9453df7651
https://towardsdatascience.com/how-to-use-dataset-in-tensorflow-c758ef9e4428
Thank you for reading,