Building scalable and efficient ML Pipelines
By: Vimarsh Karbhari
Tags: Machine Learning, Artificial Intelligence, Kubernetes, Interview, Data Science

Building scalable and efficient ML Pipelines

Using Kubernetes to build ML pipelines that scale

Kubernetes is the gold standard for managing tons of containerized applications, whether they are in the cloud or on your own hardware. Whether it is pipeline building, model building or ML application building Kubernetes enable containerization which is a safe way to build and scale any of these scenarios.

Kubernetes can host several packaged and pre-integrated data and data science frameworks on the same cluster. These are usually scalable or they auto-scale, and they’re defined/managed with a declarative approach: specify what your requirements are and the service will continuously seek to satisfy them, which provides resiliency and minimizes manual intervention.

KubeFlow is an open source project that groups leading relevant K8 frameworks. KubeFlow components include Jupyter notebooks, KubeFlow Pipeline (workflow and experiment management), scalable training services (utilized for TensorFlow, PyTourch, Horovod, MXNet, Chainer) and model serving solutions. KubeFlow also offers examples and pre-integrated/tested components.

In addition to typical data science tools, Kubernetes can host data analytics tools such as Spark or Presto, various databases and monitoring/logging solutions such as Prometheus, Grafana and Elastic Search as well. It also enables the use of serverless functions (i.e. auto built/deployed/scaled code like AWS Lambda) for a variety of data-related tasks or APIs/model serving.

The key advantage of Kubernetes vs proprietary cloud or SaaS solutions is that its tools are regularly added and upgraded, Google’s search and Stack overflow are often the fastest path to help and the solution can be deployed everywhere (any cloud service or even on-prem or on your own laptop). A community project also forces associated components/services to conform to a set of standards/abstractions which simplify interoperability, security, monitoring which in turn benefits everyone.

Bringing efficiency to ML pipelines

Unfortunately, just betting on a credible platform like Kubernetes is not enough. Life for data and engineering gets easier once we adopt three guiding rules:

Optimize for functionality — Create reusable abstract functions/steps which can accept parameters. Build for scalability— Apply parallelism to every step (or as often as possible, within reason). Automation — Avoid manual and repetitive tasks by using declarative semantics and workflows.

The current trend in data science is to build “ML factories,” similar to agile software development, build automated pipelines which take data, pre-process it, run training, generate, deploy and monitor the models. The declarative and automated deployment and scaling approach offered by Kubernetes is a great baseline, but it’s missing a way to manage such pipelines on top of that.

A relatively new tool which is part of the KubeFlow project is Pipelines, a set of services and UI aimed at creating and managing ML pipelines. We can write our own code or build from a large set of pre-defined components and algorithms contributed by companies like Google, Amazon, Microsoft, IBM or NVIDIA.

Kubeflow Piplines UI

Once we have a workflow, we can run it once, at scheduled intervals, or trigger it automatically. The pipelines, experiments and runs are managed, and their results are stored and versioned. Pipelines solve the major problem of reproducing and explaining our ML models. It also means we can visually compare between runs and store versioned input and output artifacts in various object/file repositories.

A major challenge is always running experiments and data processing at scale. Pipelines orchestrate various horizontal-scaling and GPU-accelerated data and ML frameworks. A single logical pipeline step may run on a dozen parallel instances of TensorFlow, Spark, or Nuclio functions. Pipelines also have components which map to existing cloud services, so that we can submit a logical task which may run on a managed Google AI and data service, or on Amazon’s SageMaker or EMR.

KubeFlow and its Pipelines, like most tools in this category, are still evolving, but it has a large and vibrant multi-vendor community behind it. This guarantees a viable and open framework. Much like the first days of Kubernetes, cloud providers and software vendors had their proprietary solutions for managing containers, and over time they’ve all given way to the open source standard demanded by the community.