A Kubernetes architecture for machine learning web-application deployments | by Geoffroy Gobert | Towards Data Science

Kubernetes became the reference for container orchestration. Container orchestration means starting containers, shutting them down, scaling them vertically (quantity of memory and CPU attributed), and scaling them horizontally (number of containers running in parallel).
Does Kubernetes add value to machine learning? Machine Learning needs a lot of resources for training a model, but a little resource for serving prediction. Kubernetes adapts the resources automatically! Machine learning usually implies a long training duration, that can be shortened by distributing the task over several computers. Kubernetes shines in adapting the number of computers necessary. In general, Kubernetes reduces infrastructure costs by requiring only the necessary resources over time.
This article describes a Kubernetes architecture for machine-learning web-application. The term “Kubernetes architecture” answers the question “how is the application deployed into Kubernetes?”. This article answers this question in 4 parts that:
Define a machine learning web-application.Analyze the requirements for deploying the machine learning web-application to production.Suggest a Kubernetes architecture for deploying machine learning web-application.Adapt the architecture for distributed training.
Define a machine learning web-application.
Analyze the requirements for deploying the machine learning web-application to production.
Suggest a Kubernetes architecture for deploying machine learning web-application.
Adapt the architecture for distributed training.
If you are interested in this article, you might also be interested in:
towardsdatascience.com
On a high level, a machine learning application is responsible for two things:
training a model: based on a training dataset, it trains an estimatorserving predictions: based on the trained estimator and new data, it predicts new values
training a model: based on a training dataset, it trains an estimator
serving predictions: based on the trained estimator and new data, it predicts new values
One example: demand-forecaster
Imagine an application that forecasts the demand of scooters in an area of a city. Its inputs and outputs are in red. In the next step, each input and output will be mapped to its Kubernetes component.
Input: It reads the historical demand from a database.
Output: It serves the predictions with a REST API. Uvicorn is the webserver.
Serialization: It writes and reads the trained estimator on the hard drive. This is the output of the training phase, and it is the input of the prediction phase.
Requirements for deploying machine learning web-applications:
Ensure consistency between the code and the estimator. Changing the estimator library (e.g. version of sklearn, xgboost, tensorflow) might require re-training the model. Adding a feature, updating a feature, or deleting a feature also requires re-training the model. And because a web-application is constantly serving prediction, discrepancies between the deployed code and the estimator must be avoided.Avoid model drift. A model trained yesterday will be less accurate tomorrow. This is called model drift. It matters more in some areas than in others. In forecasting, the latest observation matters, so the model is drifting fast. On the opposite, in computer vision, if the task is about to classify different types of cars, the model drifts when new models of cars are available. In order to be more accurate, model drift must be avoided.Scaling vertically. Training a model requires a lot of memory. To avoid out-of-memory errors, a sufficient quantity of memory must be provided.Scaling horizontally. Training machine learning models can be very long. Distributing the training over several computers helps. A sufficient number of computers must be provided.Share trained estimator. The model’s estimator is trained on a computer and predictions are served on another. The predictions can also be served by several computers. The serialized estimator must be shared between all the computers.
Ensure consistency between the code and the estimator. Changing the estimator library (e.g. version of sklearn, xgboost, tensorflow) might require re-training the model. Adding a feature, updating a feature, or deleting a feature also requires re-training the model. And because a web-application is constantly serving prediction, discrepancies between the deployed code and the estimator must be avoided.
Avoid model drift. A model trained yesterday will be less accurate tomorrow. This is called model drift. It matters more in some areas than in others. In forecasting, the latest observation matters, so the model is drifting fast. On the opposite, in computer vision, if the task is about to classify different types of cars, the model drifts when new models of cars are available. In order to be more accurate, model drift must be avoided.
Scaling vertically. Training a model requires a lot of memory. To avoid out-of-memory errors, a sufficient quantity of memory must be provided.
Scaling horizontally. Training machine learning models can be very long. Distributing the training over several computers helps. A sufficient number of computers must be provided.
Share trained estimator. The model’s estimator is trained on a computer and predictions are served on another. The predictions can also be served by several computers. The serialized estimator must be shared between all the computers.
Deploying a machine learning web-application on production adds new requirements. This section presents how they can be solved with Kubernetes.
Kubernetes terminology
Previously, this article spoke about “computers”, in Kubernetes it translates to “pods”.
Pods are the smallest, most basic deployable objects in Kubernetes. A Pod represents a single instance of a running process in your cluster. Pods contain one or more containers, such as Docker containers.
Kubernetes Engine Documentation
But to make things simpler: in this article, each pod will always have one container. This article speaks also about Kubernetes services:
A Kubernetes service is a logical abstraction for a deployed group of pods in a cluster (which all perform the same function). Since pods are ephemeral, a service enables a group of pods, which provide specific functions (web services, image processing, etc.) to be assigned a name and unique IP address (clusterIP).
In this example, Kubernetes will have one service. This service is responsible for forecasting the demand. It will do 2 things: training a machine learning model and serving predictions.
A Kubernetes architecture for machine learning web-applications
Kubernetes job (aka pod-init)
A Kubernetes job is a pod that is executed before the actual deployment, before that the pod of the web server starts. Kubernetes jobs can be used to train the model. So, the consistency between the code and the estimator can be ensured.
Kubernetes cronJob
A Kubernetes cronJob is a pod scheduled for being executed on a regular basis. It can be used to retrain the model on a regular basis (e.g. every day) with the latest data. So, the drift of the model can be avoided.
Kubernetes shared volume
A Kubernetes shared volume shares files between pods. So, the trained estimator can be shared between the pods.
On the schema above, the serialized models are actually saved on AWS S3, but a pure Kubernetes approach is to use shared volumes.
Kubernetes vertical pod autoscaler
A Kubernetes vertical pod autoscaler allocates the required memory and CPU to the pods dynamically over time. So, out-of-memory errors can be avoided and vertical scaling can be ensured.
Horionztal scaling
This requirement is met by 2 components. The number of deployment pods is controlled by Kubernetes horizontal pod autoscaler. It starts and stops pods in order to satisfy the number of simultaneous HTTP-requests. Besides, Kubernetes jobs have a parameter “parallelism” to control how many pods run in parallel. Jobs are used for training the model, and beyond Kubernetes, training a machine learning model in a distributed way requires some adaptation. This is the focus of the next section.
Training the model during deployment makes deployment easier. Contributors don’t need to think: How this new feature will impact the legacy estimator? How will the new normalization of the data impact the legacy estimator? How will the update of the library impact the legacy estimator?
Nonetheless, it has one downside: training the model during deployment (in the pod-initialization step) might last for a long time and make the deployment workflow (CI/CD) inefficient. A slow deployment pipeline is breaking the CI/CD best practices.
In this case, distributing the training over several pods helps. Some libraries are designed to distribute the training of one model: for example MLlib or Tensorflow. This is not the focus of this article.
In other cases, the training is fitting several models, not only one. In this case, it’s easy to distribute the training with a master/worker pattern. Assuming the entry points are bin/master.py and bin/worker.sh, this is a representation of the pods that are orchestrated by Kubernetes:
The light green boxes show the kind of Kubernetes object that is manipulated: they group pod definitions by type. The middle green boxes are the definition of the pod i.e. the YAML like below. The dark green boxes show actual instances of the definition.
apiVersion: batch/v1beta1kind: CronJobspec:  schedule: "40 2 * * *"
In a distributed world, Kubernetes orchestrates multiple instances of the pods.
For jobs and cronJobs, first, the pod named “master” is executed: it creates the task in the database (redis, outside of Kubernetes). Then, the workers are starting: they perform the task. The worker processes need to be terminated so that Kubernetes stop the pods and then starts the deployment pods. With celery, the following pseudo-code helped:
Deploying a machine learning web-service adds new requirements like ensuring the consistency between the code and the estimator, scalability, sharing the models between all computers, and avoid model-drift. This article shows one way of solving these requirements with Kubernetes. Besides, it adds great value for reducing infrastructure costs.