A Practical Guide for Deploying Embedding-Based Machine Learning Models
By: Sven Degroote
Tags: Kubernetes, Kubeflow Pipelines, Machine Learning, Production, Recommendation System

Orchestrating the model deployment with Kubeflow Pipelines

The tools and frameworks are identified to be able to perform a model deployment. Now we will need to define all the steps that will be required to execute a reliable model deployment. The required steps are, from a high-level point of view:

Deploy the new model version next to the current one. Direct streaming updates to both model versions. Start bulk load in order to backfill the new model version with all the re-calculated historical records. Once the bulk load has been processed by the new model version, it is ready to receive traffic and we can gradually shift traffic from the old to the new model. Remove the old model version

Figure 9. Steps to perform the model deployment.

Kubeflow Pipelines is an orchestrator that lets you define a set of operations and their order of execution. Each operation is defined by a Docker container. It is also possible to transfer information from one operation to another, this is extremely helpful if you have a dependent operation that needs the output of one of the previous operations. This is often the case with Machine Learning related workflows, where for example you pass on the storage bucket path to the preprocessed data to your training operation.

Ramped Model Deployment

The least complicated model deployment is to gradually ramp traffic from one model version to another in a fully automated fashion by defining a step-wise function that shifts X % every Y seconds. The speed at which you shift traffic highly depends on the volume of your traffic as you want to properly let the new model version scale in order to meet the traffic demand it gets.

Figure 10. Traffic gradually shifted from version 1 to version 2 in a fully-automated manner.

If all the deployment steps mentioned above are properly containerized, it is possible to chain this set of operations together in an execution graph so that the deployment can be reliably orchestrated. We can inspect the execution graph:

Figure 11: High-level orchestration pipeline to deploy the new model in a ramped manner.

In the ‘ramp’ step, the target weight percentages are updated so that traffic is gradually ramped to the other model version, as visualized in figure 10.

A/B Model Deployment

The steps 4 and 5 are slightly different for the A/B model deployment scenario as the model version to be promoted is variable. In this scenario, a pause step is introduced during the ramp. Based on the model performance, new target weight percentages are defined and the ramping interactively continues until a model is promoted. Traditionally in A/B testing, the new model version will only receive a small portion of the traffic at first and it will be checked how the new model version responds to real traffic. Depending on these results, the traffic might be gradually increased if the new model appears to be better. For example, this is a scenario where after a 50/50% traffic split it was decided to rollback to model version 1:

Figure 12. A/B model deployment scenario where version 1 is rolled back after splitting traffic all the way up to 50%/50%.

A simple tool to input these target weight percentages could be a message queue like Google Cloud Pub/Sub. The pipeline simply reads messages from a message queue and deploys the target weight percentages. In this case, the execution graph will have a ‘promote’ and ‘rollback’ branch, depending on the results of the A/B testing.

Figure 13. High-level orchestration pipeline to deploy a new model in A/B mode.

Conclusion

In order to benefit from the wide range of available Machine Learning Models, we need to be able to deploy them in production applications. Embedding-based models are particularly hard to deploy since all embeddings in the system need to be recalculated for each model version and traffic needs to be routed correctly while doing the model update.

We described a concrete set of tools and frameworks that can be used to implement the deployment strategy for these embedding-based models. Kubernetes makes it possible to deploy our embedding-based system, configure scaling behavior and expose services. Istio unlocks fine-grained traffic routing control mechanisms that are required to correctly split and route traffic across the model versions. Kubeflow Pipelines allows us to execute the set of operations, that are required to deploy a new model version, in a reliable and reproducible manner.

Mastering the technique for crafting model deployment pipelines as ‘a push on the button’ will unlock an immense potential. You will be able to test new machine learning models swiftly, without pain. Your engineering team does not need to spend time on redoing manual deployments over and over again. Instead, model deployments are automated, reliable, reproducible and can be triggered with a simple push on a button. This practice is often referred to as MLOps, analogous to DevOps.

About ML6

We are a team of AI experts and the fastest growing AI company in Belgium. With offices in Ghent, Amsterdam, Berlin and London, we build and implement self learning systems across different sectors to help our clients operate more efficiently. We do this by staying on top of research, innovation and applying our expertise in practice. To find out more, please visit www.ml6.eu