Ignite the Spark!
By: Alexander Sack
Tags: Big Data, Pyspark, Apache Spark, Data Science, Kubernetes

Ignite the Spark!

Source: Free-Photos from Pixabay

Introduction

In this primer, you are first going to learn a little about how Apache Spark’s cluster manager works and then how you can run PySpark within a Jupyter notebook interactively on an existing Kubernetes (k8s) cluster.

After completing this article, you should be able to develop Spark applications on any Kubernetes cluster with confidence as well as have a deeper understanding of how the two are wired together.

It’s all about context

Source: Apache Documentation

Spark is a fast and general-purpose cluster computing system which means by definition compute is shared across a number of interconnected nodes in a distributed fashion.

But how does Spark actually distribute a given workload across a cluster?

Spark adopts a Master/Slave approach whereby a driver program (“the master”) creates a SparkContext object that connects to a cluster manager.

The SparkContext created (either programmatically by your favorite language binding or on your behalf when you submit a job) abstracts the cluster as one large compute node that your application uses to perform work.

The cluster manager on the other hand, is responsible for spawning and managing a number of worker nodes (“the slaves”) that each run an executor process on behalf of the SparkContext to do the actual work.

When data in the form of a Resilient Distributed Dataframe (RDD) is manipulated by your Spark application, the RDD is split into a number of partitions and distributed across these worker node/executor combinations for processing. The final result is aggregated across the nodes and sent back to the driver.

Pick a manager, any cluster manager

One of the key advantages of this design is that the cluster manager is decoupled from your application and thus interchangeable.

Traditionally, Spark supported three types of cluster managers:

Standalone

Apache Mesos

Hadoop YARN

The Standalone cluster manager is the default one and is shipped with every version of Spark. It is a no frills, competent manager that is meant to get you up and running as fast as possible.

Apache Mesos is a clustering technology in its own right and meant to abstract away all of your cluster’s resources as if it was one big computer. Mesos ships with a cluster manager that you can leverage with Spark.

Hadoop YARN (“Yet Another Resource Negotiator”) was developed as an outgrowth of the Apache Hadoop project and mainly focused on distributing MapReduce workloads. As a result, it too is a cluster manager which Spark can talk to natively.

Enter Kubernetes

Source: Apache Documentation

As of 2.3.0, Spark now supports using Kubernetes directly as a cluster manager.

But what does that really mean?

That depends based on how you want to run Spark on Kubernetes.

In cluster mode, after you submit an application using spark-submit, the SparkContext created on behalf of your application will ask the kube-apiserver to setup a driver node and a number of corresponding worker nodes (Pods) and proceed to run your workload on top of them.

Once all the data processing has finished, on tear down, the ephemeral worker node Pods will be terminated automatically but the driver node Pod will remain so you can inspect any logs before manually deleting it.

In client mode, you create the Spark driver node as a Pod then create a SparkContext using your favorite language’s API bindings before finally submitting work.

This primer is going to focus on configuring client mode for a few reasons:

The majority of documentation available on the web is in cluster mode

The steps to submit an application using spark-submit are not that different than using the existing cluster managers

Most Spark developers will want to use client mode so they can develop new applications and explore data interactively

Prerequisites

The below instructions assume the following:

You have administrative access to either a Kubernetes cluster or to a dedicated namespace within that cluster

I’m running a small 1.17.2 eight node cluster that was setup using the kubeadm tool. I believe any cluster 1.15+ should work just fine.

If you don’t have a cluster to work with, I highly suggest installing MiniKube on your desktop as a way to follow along.

You are somewhat familiar with Kubernetes core concepts such as Pods, Deployments, and Services

You have Docker installed, understand the concept behind containers, and are comfortable building a custom image using a Dockerfile

You have access to a Docker registry either on-premise, in-cloud, or off-premise such as Docker Hub.

You are using the latest stable release of Spark which is v2.4.4 at the time of this writing

Cluster Setup

Before we start building images and deploying Pods, let’s setup a dedicated namespace where our combo Jupyter notebook/Spark driver node will be deployed as well as all of its corresponding worker nodes.

$ kubectl create namespace spark

namespace/spark created

In order for your driver Pod to request resources from the cluster it has to use some kind of ServiceAccount that has the right Role Base Access Control (RBAC).

The Spark documentation suggests creating a RoleBinding or a ClusterRoleBinding to accomplish this. Which method you choose to implement is entirely based on the security policy of your cluster.

For this example, we will create a ServiceAccount called spark in the spark namespace with a ClusterRoleBinding that gives this account the right permissions:

$ kubectl create serviceaccount spark -n spark

serviceaccount/spark created $ kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=spark:spark --namespace=spark

clusterrolebinding.rbac.authorization.k8s.io/spark-role created

Docker Setup

As per above, since every worker node is really just a Pod on our cluster, we will need a docker image with the correct Spark run-time.

Fortunately, the Spark team provides a bunch of Dockerfiles and a tool (docker-image-tool.sh) to create images with them that are ready to be deployed on any Kubernetes cluster.

Each Dockerfile is meant to be used with the three major language bindings Spark supports — Scala, Python, and R.

Let’s build the default images so we have at least some base image to work with:

$ wget -qO- http://mirrors.gigenet.com/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz | tar -xzf - $ cd spark-2.4.4-bin-hadoop2.7 && bin/docker-image-tool.sh build

Sending build context to Docker daemon 262.2MB

...

Successfully built 02fb36ac3ee0

Successfully tagged spark:latest

Sending build context to Docker daemon 262.2MB

...

Successfully built 0d33be47d094

Successfully tagged spark-py:latest

Sending build context to Docker daemon 262.2MB

...

Successfully built dc911ac3678e

Successfully tagged spark-r:latest

After the docker-image-tool.sh script completes, you should have three new images ready for deployment on your Kubernetes cluster:

$ docker images spark:latest

REPOSITORY TAG IMAGE ID CREATED SIZE

spark latest 02fb36ac3ee0 7 days ago 344MB $ docker images spark-py:latest

REPOSITORY TAG IMAGE ID CREATED SIZE

spark-py latest 0d33be47d094 7 days ago 432MB $ $ docker images spark-r:latest

REPOSITORY TAG IMAGE ID CREATED SIZE

spark-r latest dc911ac3678e 7 days ago 539MB

Since this tutorial is going to focus on using PySpark, we are going to use the spark-py image for our worker Pod. Normally, you would just push these images to whatever docker registry your cluster uses. However, we are going to create custom versions of them in order to work around a bug.

Communication Breakdown

At the time of this writing, there is an issue with using the existing Spark v2.4.4 Kubernetes client jar files with newer clusters due to a security CVE that was patched recently.

You can read more about the issue here and here.

The workaround is to build a Spark enabled docker image but with newer client jar files in it.

Let’s use the base images we just built to do exactly that. Since we are focused on using PySpark, we will only rebuild the spark-py image but the same steps apply to all of them.

$ mkdir my-spark-py && cd my-spark-py $ wget -qO- http://apache.mirrors.hoobly.com/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop2.7.tgz | tar -xzf - $ cp spark-3.0.0-preview2-bin-hadoop2.7/jars/kubernetes*jar . && rm -f spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop2.7.tgz $ cat << EOF > Dockerfile

FROM spark-py:v2.4.4 COPY *.jar /opt/spark/jars/

RUN rm /opt/spark/jars/kubernetes-*-4.1.2.jar

EOF $ docker build --rm -t my-spark-py:v2.4.4 .

...

Successfully built 29abceec5cb2

Successfully tagged my-spark-py:v2.4.4

So now my-spark-py:v2.4.4 contains the same official Spark run-time but with updated Kubernetes client jar files taken from the v3.0.0-preview2 release.

This image will be pulled by Kubernetes every time it creates a worker Pod on demand when we generate our SparkContext.

Creating the “Driver” Pod Image

Since our ultimate goal is to interactively develop a PySpark application on Kubernetes within a Jupyter notebook, that Pod will also act as our “driver” node.

Consequently, you have two options:

The official PySpark docker image is a great place to start and I highly recommend it.

I built a custom docker image that you can find by clicking the link above. I call this image my-notebook:latest.

You should build, tag and push whatever image you ultimately choose to the docker registry that your cluster uses.

Setting up your Deployment

As is all things Kubernetes, we need to write a YAML file to deploy our driver Pod that will be used to log into Jupyter and create a SparkContext.

Here is what my deployment looks like:

apiVersion: apps/v1

kind: Deployment

metadata:

namespace: spark

name: my-notebook-deployment

labels:

app: my-notebook

spec:

replicas: 1

selector:

matchLabels:

app: my-notebook

template:

metadata:

labels:

app: my-notebook

spec:

serviceAccountName: spark

containers:

- name: my-notebook

image: pidocker-docker-registry.default.svc.cluster.local:5000/my-notebook:latest

ports:

- containerPort: 8888

volumeMounts:

- mountPath: /root/data

name: my-notebook-pv

workingDir: /root

resources:

limits:

memory: 2Gi

volumes:

- name: my-notebook-pv

persistentVolumeClaim:

claimName: my-notebook-pvc

---

apiVersion: v1

kind: Service

metadata:

namespace: spark

name: my-notebook-deployment

spec:

selector:

app: my-notebook

ports:

- protocol: TCP

port: 29413

clusterIP: None

The highlighted text are directives you should pay attention to:

We want our deployment to live in the spark namespace

namespace We want to set this deployment’s ServiceAccount to spark which has the ClusterRoleBinding spark-role we created above

to which has the ClusterRoleBinding we created above We want to expose our deployment as a headless service as per the Spark client mode documentation by setting the service’s ClusterIP to None. This allows the worker Pods to communicate with the driver Pod via port 29413. The port number you choose can be any valid unused port but I choose 29413 in this example as an homage to this helpful post.

Create the Deployment

When you are satisfied with your deployment, create it via:

$ kubectl create -f k8s/deploy.yaml

deployment.apps/my-notebook-deployment created

service/my-notebook-deployment created

Check that your deployment is running:

$ kubectl get all -n spark

NAME READY STATUS RESTARTS AGE

pod/my-notebook-deployment-7bf574447c-pdn2q 1/1 Running 0 19s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/my-notebook-deployment ClusterIP None <none> 29413/TCP 19s NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/my-notebook-deployment 1/1 1 1 19s NAME DESIRED CURRENT READY AGE

replicaset.apps/my-notebook-deployment-7bf574447c 1 1 1 19s

And now port-forward to 8888 so you can login into Jupyter:

$ kubectl port-forward -n spark deployment.apps/my-notebook-deployment 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888

Forwarding from [::1]:8888 -> 8888

Now you should be able to point your favorite browser to http://localhost:8888 and log into Jupyter (if you used my image as a reference the password is ‘jupyter’ else you will need to copy/paste the generated auth token displayed in the container’s log file).

Light It!

Start a new Python3 notebook and type the following in a cell:

import os from pyspark import SparkContext, SparkConf

from pyspark.sql import SparkSession

sparkConf = SparkConf()

sparkConf.setMaster("k8s://https://kubernetes.default.svc.cluster.local:443")

sparkConf.setAppName("spark")

sparkConf.set("spark.kubernetes.container.image", "pidocker-docker-registry.default.svc.cluster.local:5000/my-spark-py:v2.4.4")

sparkConf.set("spark.kubernetes.namespace", "spark")

sparkConf.set("spark.executor.instances", "7")

sparkConf.set("spark.executor.cores", "2")

sparkConf.set("spark.driver.memory", "512m")

sparkConf.set("spark.executor.memory", "512m")

sparkConf.set("spark.kubernetes.pyspark.pythonVersion", "3")

sparkConf.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")

sparkConf.set("spark.kubernetes.authenticate.serviceAccountName", "spark")

sparkConf.set("spark.driver.port", "29413")

sparkConf.set("spark.driver.host", "my-notebook-deployment.spark.svc.cluster.local") # Create Spark config for our Kubernetes based cluster managersparkConf = SparkConf()sparkConf.setMaster("sparkConf.setAppName(sparkConf.set(sparkConf.set(sparkConf.set(sparkConf.set("spark.executor.cores", "2")sparkConf.set("spark.driver.memory", "512m")sparkConf.set("spark.executor.memory", "512m")sparkConf.set("spark.kubernetes.pyspark.pythonVersion", "3")sparkConf.set(sparkConf.set(sparkConf.set(")sparkConf.set(") # Initialize our Spark cluster, this will actually

# generate the worker nodes.

spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()

sc = spark.sparkContext

Let’s break down the Spark config above:

We set the master URL to “ k8s:// https://kubernetes.default.svc.cluster.local:443 " which tells Spark that our cluster manager is Kubernetes (k8s) and where to find the kube-apiserver to request resources

which tells Spark that our cluster manager is Kubernetes (k8s) and where to find the kube-apiserver to request resources We set the application name to “spark” which will be prepended to the worker Pod names. Feel free to use whatever name you deem appropriate

which will be prepended to the worker Pod names. Feel free to use whatever name you deem appropriate We set the container image to use for the worker Pods to the custom one we built above that has the updated Kubernetes client jar files in it

We specify that we want all worker nodes to be created in the spark namespace

namespace I set the number of worker nodes to “7” since I am on eight node closer (one node is the master, seven are available for work). You need to pick a number that makes sense for your cluster.

since I am on eight node closer (one node is the master, seven are available for work). You need to pick a number that makes sense for your cluster. We set the service account names for both the driver and workers to our ServiceAccount wecreated above ( “spark” )

) Finally, we set the port number to 29413 and the fully qualified domain name of my in-cluster service which specifies to the worker nodes where to contact the driver node.

After you execute this cell, you should see a number of worker Pods being spawned in the spark namespace.

Let’s check it:

$ kubectl get pods -n spark

NAME READY STATUS RESTARTS AGE

my-notebook-deployment-7bf574447c-pdn2q 1/1 Running 0 5m6s

spark-1581043649282-exec-1 1/1 Running 0 76s

spark-1581043651262-exec-2 1/1 Running 0 75s

spark-1581043651392-exec-3 1/1 Running 0 75s

spark-1581043651510-exec-4 1/1 Running 0 75s

spark-1581043651623-exec-5 1/1 Running 0 75s

spark-1581043656753-exec-6 1/1 Running 0 70s

spark-1581043656875-exec-7 1/1 Running 0 70s

You now have an a Spark cluster running on top of Kubernetes waiting for work!

As is tradition, let’s calculate the number Pi using our cluster in a new cell:

from random import random

from operator import add partitions = 7

n = 10000000 * partitions def f(_):

x = random() * 2 - 1

y = random() * 2 - 1



return 1 if x ** 2 + y ** 2 <= 1 else 0 count = sc.parallelize(range(1, n + 1), partitions).map(f).reduce(add)

print("Pi is roughly %f" % (4.0 * count / n))

Your output should be:

Pi is roughly 3.141397

Extinguish the Spark

After you have completed your development, you can shutdown your Spark cluster by simply executing:

sc.stop()

That will instruct your SparkContext to tear down the worker Pods you created above via the kube-apiserver.

To verify:

$ kubectl get pods -n spark

NAME READY STATUS RESTARTS AGE

my-notebook-deployment-7bf574447c-pdn2q 1/1 Running 0 20m

spark-1581043649282-exec-1 0/1 Terminating 0 16m

spark-1581043651262-exec-2 1/1 Terminating 0 16m

spark-1581043651392-exec-3 1/1 Terminating 0 16m

spark-1581043651510-exec-4 0/1 Terminating 0 16m

spark-1581043651623-exec-5 0/1 Terminating 0 16m

spark-1581043656753-exec-6 0/1 Terminating 0 16m

spark-1581043656875-exec-7 0/1 Terminating 0 16m

And now you are left with just your deployment/driver Pod in the spark namespace.

Conclusion

I hope you are now more confident creating a Spark cluster on Kubernetes using PySpark and appreciate the architecture behind it.

I know I used to run in stand alone mode for most of my early stage Spark development work, but now with Kubernetes support, it’s cluster or bust!

Feel free to post any questions, corrections, or concerns below and Happy Spark’ing!