How to Serve Different Model Versions using TensorFlow Serving | by Renu Khandelwal | Towards Data Science

This article explains how to manage multiple models and multiple versions of the same model in TensorFlow Serving using configuration files along with a brief understanding of batching.
Deploying a TensorFlow Model to Production made Easy
You have TensorFlow deep learning models with different architectures or have trained your models with different hyperparameters and would like to test them locally or in production. The easiest way is to serve the models using a Model Server Config file.
A Model Server Configuration file is a protocol buffer file(protobuf), which is a language-neutral, platform-neutral extensible yet simple and faster way to serialize the structure data.
How do I create a Model Config File?
Below is a sample Model Config File that will serve all the versions of the “MNIST” models available on disk.
model_config_list { config {    name: 'mnist'    base_path: '/models/mnist/'    model_platform: 'tensorflow'    model_version_policy: {all: {}}        }}
Each ModelConfig specifies one model to be served with the following parameters.
name - A servable model name
base_path - Specify the path to look for versions of the servable.
model_platform - Platform in which the model was developed.
model_version_policy - Version policy for the model indicates which version of the model to load and serve the client. By default, the latest version of the model will be served and can be overridden by changing the model_version_policy field.
How to load the Model Config File?
Read how to load the TensorFlow Serving Docker container on Windows 10 here
docker run -p 8501:8501 --mount type=bind,source=C:\TF_serv\TF_model,target=/models/mnist -e MODEL_NAME=mnist -t tensorflow/serving
List all the running Docker containers
docker container list
or
docker ps
Copy the models.config file from the source to the target of the Docker image
docker cp \TF_serv\TF_model\models.config ba978f5a9475:/models/mnist
When copying the file to docker container, provide the container Id, as shown above.
Stop the Docker container using the Docker container name or container Id
docker stop ba978f5a9475
Finally, loading the Model Config file using the option
--model_config_file_poll_wait_seconds flag instructs the server to check for a new config file at the path specified using --model_config_file every 60 seconds.
docker run -p 8501:8501 --mount type=bind,source=C:\TF_serv\TF_model,target=/models/mnist -e MODEL_NAME=mnist -t tensorflow/serving --model_config_file_poll_wait_seconds=60 --model_config_file=/models/mnist/models.config
How to make an inference using the model served using models.config file using a specific version of the model?
Using REST API to make the inference request
json_response = requests.post('http://localhost:8501/v1/models/mnist/versions/5:predict', data=data, headers=headers)
Since the models.config file specified above will load all versions on the disk; I have loaded version 5.
A generic way to call a specific version of the model to make an inference is shown below.
/v1/models/<model name>/versions/<version number>
How do I override and specify a specific version of the model to be loaded and served?
You will need to use the specified tag with the version number within the model_version_policy
model_config_list { config {    name: 'mnist'    base_path: '/models/mnist/'    model_platform: 'tensorflow'    model_version_policy{                        specific{                           versions:2                                }                         } }}
with the above change models.config file, the client can now call the model’s predict method only for version 2.
Can I server multiple versions of the same model simultaneously?
You can also serve multiple versions of the same model simultaneously. In the example below, we are serving the MNIST model with versions 2 and 3 simultaneously.
model_config_list { config {    name: 'mnist'    base_path: '/models/mnist/'    model_platform: 'tensorflow'    model_version_policy{                        specific{                           versions:2                           versions:3                              }                         } }}
This is helpful when you have a newer version of the model and would like to divert a few users to a newer version and keep most clients on the stable older version of the model. This allows you to accomplish the A/B testing.
Can I server multiple models like a custom model and a transfer learned model?
To server multiple models, update the models.config file as shown below
model_config_list {  config{         name: 'mnist'         base_path: '/models/mnist/'         model_platform: 'tensorflow'         model_version_policy{             specific{                      versions:2                      versions:3                      }                    }       }  config{        name: 'Inception_1'        base_path: '/models/inception_1/'        model_platform: 'tensorflow'        }}
Will TensorFlow Serving allow for batching multiple requests?
TensorFlow Serving allows two different forms of batching.
Batch individual model inference requests, where TensorFlow serving waits for a predetermined time and then perform inferences on all requests that arrived in that time period
A single client can send batched requests to TensorFlow Serving.
Batching requires all requests to use the same version of the model.
Batching allows for higher resource utilization and higher throughput.
To enable batching, specify --enable_batching flag as true and also set the config file containing batching parameters using --batching_parameters_file flag.
docker run -p 8501:8501 --mount type=bind,source=C:\TF_serv\TF_model,target=/models/mnist -e MODEL_NAME=mnist -t tensorflow/serving --model_config_file_poll_wait_seconds=60  --model_config_file=/models/mnist/models.config --enable_batching=true --batching_parameters_file=/models/mnist/batch.config
The batch config file is a .protobuf file
max_batch_size { value: 128 }batch_timeout_micros { value: 0 }max_enqueued_batches { value: 1000000 }num_batch_threads { value: 8 }
max_batch_size: allows you to specify the maximum size of any batch. If you specify a batch larger than the max_batch_size, then you will get an error. This parameter governs the throughput/latency tradeoff and ensures the resource constraints are not exceeded.
batch_timeout_micros: This is the maximum amount of time for the server to retry the batched request. The parameter value is specified in microseconds.
num_batch_threads: defines the degree of parallelism by specifying the maximum number of batches to be processed concurrently.
max_enqueued_batches: It helps limit batch queue by turning away requests that would take a long time to be served and avoid building a large backlog of requests.
When I ran the below code with and without batching, batching gave me results almost 2 to 3 times faster.
import time#Build the batched data for making iferences for 100 imagesdata = json.dumps({"signature_name": "serving_default", "instances": test_images[:100].tolist()})st=time.perf_counter()headers = {"content-type": "application/json"}json_response = requests.post('http://localhost:8501/v1/models/mnist/labels/2:predict', data=data, headers=headers)predictions = json.loads(json_response.text)['predictions']end_t= time.perf_counter()print(end_t-st)
TensorFlow Serving has made the production deployment easier by providing easy ways to serve different models or different versions of the same model using Model Server configuration.
Batching allows you to batch multiple requests for the same client or different clients, thus optimizing the hardware accelerator resources and providing better throughput.