Serving TensorFlow models with TF Serving | by Álvaro Bartolomé | Towards Data Science

TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs. TensorFlow Serving provides out-of-the-box integration with TensorFlow models, but can be easily extended to serve other types of models and data.
Currently there are a lot of different solutions to serve ML models in production with the growth that MLOps is having nowadays as the standard procedure to work with ML models during all their lifecycle. Maybe the most popular one is TensorFlow Serving developed by TensorFlow so as to server their models in production environments.
This post is a guide on how to train, save, serve and use TensorFlow ML models in production environments. Along the GitHub repository linked to this post we will prepare and train a custom CNN model for image classification of The Simpsons Characters Data dataset, that will be later deployed using TensorFlow Serving.
So as to get a better understanding on all the process that is presented in this post, as a personal recommendation, you should read it while you check the resources available in the repository, as well as trying to reproduce it with the same or with a different TensorFlow model, as “practice makes the master”.
github.com
First of all you need to make sure that you have all the requirements installed, but before proceeding you need to know that TF-Serving is just available for Ubuntu, which means that, in order to use it you will either need a Ubuntu VM or just Docker installed in your OS so as to run a Docker container which deploys TF-Serving.
Warning! ⚠️ In case you don’t have Ubuntu, but still want to deploy TF-Serving via Docker, you don’t need to install it with APT-GET as you will not be able to, you just need to run the Dockerfile (go to the section Docker).
So, from your Ubuntu VM you need to install tensorflow-model-server, but before being able to install it you need to add the TF-Serving distribution URI as a package source as it follows:
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list && \curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
And then you will be able to install tensorflow-model-server using APT-GET as it follows:
apt-get update && apt-get install tensorflow-model-server
Finally, for the client side you need to install the Python package tensorflow-serving-api, which is useful towards using the gRPC API; and, tensorflow. Note that the versions of both packages should match.
pip install tensorflow-serving-api==2.4.1pip install tensorflow==2.4.1
If you have any problems regarding the TensorFlow installation, visit Installation | TensorFlow.
The dataset that is going to be used to train the image classification model is “The Simpsons Characters Data”, which is a big Kaggle dataset that contains RGB images of some of the main The Simpsons characters including Homer, Marge, Bart, Lisa, Maggie, Barney, and much more.
The original dataset contains 42 classes of The Simpsons characters, with an unbalanced number of samples per class, and a total of 20,935 training images and 990 test images in JPG format, and the images in different sizes, but as all of them are small, we will be resizing them to 64x64px when training the model.
Anyway, we will create a custom slice of the original dataset keeping just the training set, and using a random 80/20 train-test split and removing the classes with less than 50 images. So on, we will be have 32 classes, with 13,210 training images, 3,286 validation images, and 4,142 testing images.
Once the data has been explored, we are going to proceed with the definition of the ML model, that in this case will be a CNN (Convolutional Neural Network) as we are facing an image classification problem.
The created model architecture consists on an initial Conv2D layer (that also indicates the input_shape of the net), which is a 2D convolutional layer that produces 16 filters as output of windows of 3x3 convolutions, followed by a MaxPooling2D in order to downsample the Tensor resulting from the previous convolutional layer. Usually, you will find this layer after two consecutive convolutions, but for the sake of simplicity, here we will be downsampling the data after each convolution, as this is a simple CNN with a relatively small dataset (less than 20k images).
Then we will include another combination of Conv2D and MaxPooling2D layers as increasing the number of convolutional filters means that we will provide more data to the CNN as it is capturing more combinations of pixel values from the input image Tensor.
After applying the convolutional operations, we will include a Flatten layer in order to transform the image Tensor into a 1D Tensor which prepares the data that goes through the CNN so as to include a few fully connected layers after it.
Finally, we will include some Dense fully connected layers so as to assign the final weights of the net, and some Dropout layers to avoid overfitting during the training phase. You also need to take into consideration that the latest Dense layer contains as much units as the total labels to predict, which in this case is the number of The Simpsons Characters available in the training set.
The trained model has been named SimpsonsNet (this name will be used later while serving the model as its identifier) and its architecture looks like:
Finally, once trained we will need to dump the model in SavedModel format, which is the universal serialization format for the TensorFlow models. This format provides a language-neutral format to save ML models that is recoverable and hermetic. It enables higher-level systems and tools to produce, consume and transform TensorFlow models.
The resulting model’s directory should more or less look like the following:
assets/assets.extra/variables/    variables.data-?????-of-?????    variables.indexsaved_model.pb
More information regarding the SavedModel format at TensorFlow SavedModel.
Finally, as a personal recommendation you should check/keep an eye on the following courses:
🔥 Laurence Moroney’s TensorFlow Proffesional Certificate (previously Specialization) at Coursera for learning the basics of TensorFlow as you playaround with some common Deep Learning scenarios like CNNs, Time Series and NLP. So feel free to check it at Coursera | TensorFlow in Practice, and the course’s resources in GitHub at dlaicourse.
⭐ Daniel Bourke’s TensorFlow Zero to Mastery course he is currently developing and it will be completely free including a lot of resources. So feel free to check it in GitHub at tensorflow-deep-learning.
Once the model has been saved using SavedModel format, it is pretty straight forward to get TF-Serving working, if the installation succeeded. Unlike TorchServe, serving ML models in TF-Serving is simpler as you just need to have tensorflow-model-server installed and the ML model in the specified format.
tensorflow_model_server --port=8500 --rest_api_port=8501 \                        --model_name=simpsonsnet \                        --model_base_path=/home/saved_models/simpsonsnet
Now, even though the command is clear and self-explanatory, a more detailed explanation on the flags used is presented:
— port: this is the port to listen on for the gRPC API, the default value is 8500; but it's a common practice to still define this flag's value so as to always know the configuration of the deployed TF-Serving Server.
— rest_api_port: this is the REST API port, which is set to zero by default, which means that the REST API will not be deployed/exposed unless you manually set a port. There's no default value, it just needs to be different than the gRPC port, so we will set it to 8501 (the next port).
— model_name: this is the name of the ML model to serve, which is the one that will be exposed in the endpoint.
— model_base_path: this is the base path where the ML model that is going to be served is placed in. Note that his is the absolute path, do not use relative paths.
More information about the TF-Serving CLI available at Train and serve a TensorFlow model with TensorFlow Serving and you can also check tensorflow_model_server --help.
Once TF-Serving has been successfully deployed, you can send a sample HTTP GET request to the Model Status REST API available at http://localhost:8501/v1/models/simpsonsnet; that returns the basic information of the served ML model.
curl http://localhost:8501/v1/models/simpsonsnet
That should output something similar to the following JSON response if everything succeeded:
Note: there is not a way to gracefully shutdown server, as stated in this issue.
To shutdown the server you have two alternatives:
Getting the PID of tensorflow_model_server and killing that process:
ps aux | grep -i "tensorflow_model_server"kill -9 PID
Getting the running Docker Container ID and stopping/killing it:
docker ps # Retrieve the CONTAINER_IDdocker kill CONTAINER_ID
None of those is the recommended way of shutting down a server, as you are supposed to have a proper way to shut it down, but, currently those are the possibilities. If you have more information about this issue or a cleaner way to do this, please let me know!
In order to reproduce the TF-Serving deployment in an Ubuntu Docker image, you can use the following set of commands:
docker build -t ubuntu-tfserving:latest deployment/docker run --rm --name tfserving_docker -p8500:8500 -p8501:8501 -d ubuntu-tfserving:latest
Note: make sure that you use the -d flag in docker run so that the container runs in the background and does not block your terminal.
For more information regarding the Docker deployment, you should check TensorFlow’s explanation and notes available at TF-Serving with Docker, as it also explains how to use their Docker image (instead of a clear Ubuntu one) and some tips regarding the production deployment of the models using TF-Serving.
Along this section we will see how to interact with the deployed APIs (REST and gRPC) via Python, so as to send sample requests to classify images from The Simpsons Characters.
Regarding the REST requests to the deployed TF-Serving Prediction API you need to install the requirements as it follows:
pip install tensorflow==2.4.1pip install requests==2.25.1
And then use the following script which will send a sample The Simpsons image to be classified using the deployed model:
Now, regarding the gRPC requests to the deployed TF-Serving Prediction API you need to install the requirements as it follows:
pip install tensorflow==2.4.1pip install tensorflow-serving-api==2.4.1pip install grpcio==1.35.0
And then use the following script which will send a sample The Simpsons image to be classified using the deployed model:
To sum up, serving models with TensorFlow Serving is pretty easy thanks to the Keras SavedModel format in which the TensorFlow ML models is being dumped. TF-Serving is actively maintained by TensorFlow, which means that its usage is recommended for the LTS (Long Time Support) they provide.
Both the consistency and the ease to deploy ML models, makes TF-Serving a tool to consider in order to serve TensorFlow’s models in production environments. As the model deployment is a critical part in the MLOps lifecycle, the current ML serving solutions are a must-known if you work with the main Deep Learning frameworks (TensorFlow and PyTorch) as their serving solutions, TensorFlow Serving and TorchServe, respectively, make the ML model serving easy and scalable.
So you so highly take this into consideration when serving your ML models in production environments! ⚡
Credits for “The Simpsons Characters Dataset” to Alexandre Attia for creating it, as well as the Kaggle community that made it possible, as they included a lot of images to the original dataset (from 20 characters to up to 42, but we just used 32).
You can contact me either via Twitter or via GitHub. Or just go to AllMyLinks where you can find all my links.
🌟 Additionally you can follow me at GitHub to support me so that I keep on developing open source content! 🌟