Data pipelines on Spark and Kubernetes
By: Vimarsh Karbhari
Tags: Machine Learning, Artificial Intelligence, Spark, Data Science, Kubernetes

Data pipelines on Spark and Kubernetes

Considerations for using Apache Spark and Kubernetes to process data

If you’re running data pipelines and workflows to get data from one location to the data lake, that usually means that the team will need to process huge amounts of data. To do this in a scalable way and to handle complex computation steps across a large amount of data (effectively from a cost perspective), Kubernetes is a great choice for scheduling Spark jobs, compared to YARN.

Apache Spark is a framework that can quickly perform processing tasks on very large data sets, and Kubernetes is a portable, extensible, open-source platform for managing and orchestrating the execution of containerized workloads and services across a cluster of multiple machines.

From an architectural perspective, when you submit a Spark application, one is directly interacting with Kubernetes, the API server, which will schedule the driver pod, the Spark driver container. Then the Spark driver and the Kubernetes Cluster will talk to each other to request and launch Spark executors. This can happen statically or this can happen, dynamically if you enable dynamic application.

Dependency management

When the team uses Kubernetes, each Spark app has its own Docker image, this means the team can have full isolation and full control of the environment. The team can set their Spark version, Python version, dependencies using this architecture. These containers package the code required to execute the workload, but also all the dependencies needed to run that code, removing the hassle of maintaining a common dependency for all workloads running on a common infrastructure.

Dynamic autoscaling

Another capability with this setup is that the team can have Spark applications with dynamic allocation activated and due scanning on the cluster. This also leads to better resource management, as the scheduler takes care of picking which nodes to deploy the workloads on in combination with the fact that in the cloud, scaling a cluster up/down is quick and easy because it’s just a matter of adding or removing VMs to the cluster, and the managed Kubernetes offerings have helpers for that. This in practice is a major cost saver.

Deployment

In the hybrid cloud world today enterprises want to prevent lock-ins. Running Spark on Kubernetes means building once and deploying anywhere, which makes a cloud-agnostic approach scalable.

Metrics and Security

For metrics, the team can export everything to a time series DB. This enables the superposition of a Spark stage boundaries and the resource usage metrics.

Kubernetes is a technology which has role based access control model and secrets management. There are many open source projects that the team can leverage with which managing security is easy like the HashiCorp vault.

Finally, running Spark on Kubernetes will save the team time. Data scientists, data engineers and data architects’ time is valuable, and this setup will bring more productivity to those people and departments will could lead to savings.