Pyspark on kubernetes
Tags: Kubernetes, Spark, Azure

Introduction

Spark is a fast and general-purpose cluster computing system which means by definition compute is shared across a number of interconnected nodes in a distributed fashion.

We are going to deploy spark on AKS in client mode because pyspark seems to only support client mode.

First, we will have a quick introduce about client mode and cluster mode.

In the cluster mode, the Spark driver or spark application master will get started in any of the worker machines. The Spark driver will run the main function of application and create a so called SparkContext. SparkContext will connect to the Spark cluster manager for task schedule, resources apply and monitor. And it will also terminate the driver once the executor finishs its jobs.

In the client mode, the client who is submitting the spark application will start the driver and it will maintain the spark context. So, till the particular job execution gets over, the management of the task will be done by the driver. Also, the client should be in touch with the cluster. The client will have to be online until that particular job gets completed.

Because the pyspark only supports client mode(or as far as I know). This document will only apply the client mode.

Infrastructure

Running Pyspark inside the AKS.

2. Seperate the Pyspark driver and Spark executer

My enviorment: jupyter (pyspark+python3.8.6+java11)->AKS(spark3.0.1+python3.8.6)

You may customize your environment by building your own notebook and pyspark image. Please notice that the python and java version should be the same between driver and executor.

Cluster Setup

Before we start, we should create a serviceaccount in the cluster and grant certain access by rolebinding. I will setup all environment in spark namespace.

$ kubectl create namespace spark

namespace/spark created

In order for your driver Pod to request resources from the cluster it has to use some kind of ServiceAccount that has the right Role Base Access Control (RBAC).

The Spark documentation suggests creating a RoleBinding or a ClusterRoleBinding to accomplish this. Which method you choose to implement is entirely based on the security policy of your cluster.

For this example, we will create a ServiceAccount called spark in the spark namespace with a ClusterRoleBinding that gives this account the right permissions:

$ kubectl create serviceaccount spark -n spark

serviceaccount/spark created

$ kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=spark:spark --namespace=spark

clusterrolebinding.rbac.authorization.k8s.io/spark-role created

Docker Setup for Spark executor

This step is to build Spark executor image. This image will be used as executor in the cluster. I used Spark 3.0.1.

By default, the PySpark will use Python 3.6.8 and java 8. But I customized it to use Python 3.8.6 and java11, you can build it base on your need. By default, you can use “ ./bin/docker-image-tool.sh -r <repo> -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build ”. That command will allow you to create a default Pyspark image with 3.6.8 Python. Or you will need to write your own image Docker file. I already built my image in tarrantro/spark-py:v3.0.1, feel free to use.

You can skip this step if you want to use my image, below steps is about how to build your own image:

$ wget https://downloads.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz

$ cd spark-3.0.1-bin-hadoop2.7 $ tar -zxvf spark-3.0.1-bin-hadoop2.7.tgz $ cd spark-3.0.1-bin-hadoop2.7

$ bin/docker-image-tool.sh -r -t v3.0.1 -b java_image_tag=11-jre-slim -p Dockerfile build #You can remove "-b java_image_tag=11-jre-slim -p Dockerfile" so it will use default setting. docker push <repo>/spark-py:v3.0.1

Method 1: Build jupyter inside the cluster

As per above, this will deploy a jupyter deployment inside the AKS cluster and you may access through service. This method can save you from configuring your own environment. However, it’s less flexible.

I used offical jupyter image. If you need, you may also customize the jupyter image.

deployment.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

namespace: spark

name: my-notebook-deployment

labels:

app: my-notebook

spec:

replicas: 1

selector:

matchLabels:

app: my-notebook

template:

metadata:

labels:

app: my-notebook

spec:

serviceAccountName: spark

securityContext:

fsGroup: 100

runAsUser: 1000

containers:

- name: my-notebook

image: jupyter/pyspark-notebook

ports:

- containerPort: 8888

volumeMounts:

- mountPath: /home/jovyan/work

name: my-notebook-pv

resources:

limits:

memory: 2Gi

volumes:

- name: my-notebook-pv

persistentVolumeClaim:

claimName: my-notebook-pvc

pvc.yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: my-notebook-pvc

namespace: spark

spec:

accessModes:

- ReadWriteOnce

storageClassName: managed-premium

resources:

requests:

storage: 30Gi

service.yaml

apiVersion: v1

kind: Service

metadata:

namespace: spark

name: my-notebook-deployment

spec:

type: LoadBalancer

selector:

app: my-notebook

ports:

- protocol: TCP

port: 8888

targetPort: 8888

---

apiVersion: v1

kind: Service

metadata:

namespace: spark

name: spark-driver

spec:

selector:

app: my-notebook

ports:

- protocol: TCP

port: 29413

clusterIP: None

We want to expose our deployment as a headless service as per the Spark client mode documentation by setting the service’s ClusterIP to None. This allows the worker Pods to communicate with the driver Pod via port 29413. The port number you choose can be any valid unused port but I choose 29413 in this example as an homage to this helpful post.

Later, apply the all the yaml files.

tarrant@testubuntujp1:/data/spark$ kubectl get deployments -n spark

NAME READY UP-TO-DATE AVAILABLE AGE

my-notebook-deployment 1/1 1 1 8d

tarrant@testubuntujp1:/data/spark$ kubectl get svc -n spark

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

my-notebook-deployment LoadBalancer 10.0.178.68 20.43.95.108 8888:31002/TCP 9d

spark-driver ClusterIP None <none> 29413/TCP 9d

Now you should be able to browse to http://<external-IP>:8888 and log into Jupyter. Please paste the token from “kubectl logs <note-pod> -n spark”

Start a new Python3 notebook and type the following in a cell:



from pyspark import SparkContext, SparkConf

from pyspark.sql import SparkSession

# Create Spark config for our Kubernetes based cluster manager

sparkConf = SparkConf()

sparkConf.setMaster("k8s://

sparkConf.setAppName("spark")

sparkConf.set("spark.kubernetes.container.image", "tarrantro/spark-py:v3.0.1") #change that to your pyspark image

sparkConf.set("spark.kubernetes.namespace", "spark")

sparkConf.set("spark.executor.instances", "3")

sparkConf.set("spark.executor.cores", "2")

sparkConf.set("spark.driver.memory", "512m")

sparkConf.set("spark.executor.memory", "512m")

sparkConf.set("spark.kubernetes.pyspark.pythonVersion", "3")

sparkConf.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")

sparkConf.set("spark.kubernetes.authenticate.serviceAccountName", "spark")

sparkConf.set("spark.driver.port", "29413")

sparkConf.set("spark.driver.host", "spark-driver.spark.svc.cluster.local")

# Initialize our Spark cluster, this will actually

# generate the worker nodes.

spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()

sc = spark.sparkContext import osfrom pyspark import SparkContext, SparkConffrom pyspark.sql import SparkSession# Create Spark config for our Kubernetes based cluster managersparkConf = SparkConf()sparkConf.setMaster("k8s:// https://<AKS-API-domain-name>:443 ")sparkConf.setAppName("spark")sparkConf.set("spark.kubernetes.container.image", "tarrantro/spark-py:v3.0.1") #change that to your pyspark imagesparkConf.set("spark.kubernetes.namespace", "spark")sparkConf.set("spark.executor.instances", "3")sparkConf.set("spark.executor.cores", "2")sparkConf.set("spark.driver.memory", "512m")sparkConf.set("spark.executor.memory", "512m")sparkConf.set("spark.kubernetes.pyspark.pythonVersion", "3")sparkConf.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")sparkConf.set("spark.kubernetes.authenticate.serviceAccountName", "spark")sparkConf.set("spark.driver.port", "29413")sparkConf.set("spark.driver.host", "spark-driver.spark.svc.cluster.local")# Initialize our Spark cluster, this will actually# generate the worker nodes.spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()sc = spark.sparkContext

You may see from your AKS as below.

$ kubectl get pods -n spark

NAME READY STATUS RESTARTS AGE

my-notebook-deployment-7fb5cfbcf7-4kb98 1/1 Running 0 8d

spark-51cfd876503bc599-exec-1 1/1 Running 0 17m

spark-51cfd876503bc599-exec-2 1/1 Running 0 17m

spark-51cfd876503bc599-exec-3 1/1 Running 0 17m

test 1/1 Running 0 9d

Let’s calculate the number Pi using our cluster in a new cell:

from random import random

from operator import add

partitions = 7

n = 10000000 * partitions

def f(_):

x = random() * 2 - 1

y = random() * 2 - 1



return 1 if x ** 2 + y ** 2 <= 1 else 0

count = sc.parallelize(range(1, n + 1), partitions).map(f).reduce(add)

print("Pi is roughly %f" % (4.0 * count / n))

Method 2: Build jupyter in VM

As per above, this will deploy a jupyter in VM. This method will allow you to run your jupyter in everywhere.

Install java11.

$ sudo yum install java-11-openjdk-devel

Now, let’s install the prerequisites of Python.

$ sudo yum -y install wget gcc make libffi-devel zlib-devel readline-devel bzip2-devel ncurses-devel sqlite-devel gdbm-devel xz-devel tk-devel openssl-devel

Download Python 3.8.6 and complie



$ tar -zxvf Python-3.8.6.tgz

$ cd Python-3.8.6

$ ./configure && make && make install

$ export PYTHONPATH=$PYTHONPATH:/usr/local/bin/ $ wget https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tgz $ tar -zxvf Python-3.8.6.tgz$ cd Python-3.8.6$ ./configure && make && make install$ export PYTHONPATH=$PYTHONPATH:/usr/local/bin/

You may add “export PYTHONPATH=$PYTHONPATH:/usr/local/bin/” to ~/.bash_profile and use “source ~/.bash_profile” to enable new Python.

After finished the complie, we will install jupyter. And enable jupyter to listen on 0.0.0.0.

$ sudo pip3 install notebook

$ jupyter notebook --generate-config

Modify jupyter_notebook_config.py. You may refer to https://jupyter-notebook.readthedocs.io/en/stable/public_server.html#running-a-public-notebook-server for notebook configuration.

c.NotebookApp.ip = '*'

Install Pyspark

$ sudo pip3 install pyspark

Install kubectl

$ curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"

$ chmod +x ./kubectl

$ sudo mv ./kubectl /usr/local/bin/kubectl

Get AKS credential

$ az login

$ az aks get-credentials -g <AKS-rg> -n <AKS-name> #please replace your own AKS resource group and name.

Start jupyter.

jupyter notebook

Access the jupyter as method 1. And then, enter below code in the cell:



from pyspark import SparkContext, SparkConf

from pyspark.sql import SparkSession

# Create Spark config for our Kubernetes based cluster manager

sparkConf = SparkConf()

sparkConf.setMaster("k8s://

sparkConf.setAppName("spark")

sparkConf.set("spark.kubernetes.container.image", "tarrantro/spark-py:v3.0.1") #change that to your pyspark image

sparkConf.set("spark.kubernetes.namespace", "spark")

sparkConf.set("spark.executor.instances", "3")

sparkConf.set("spark.executor.cores", "2")

sparkConf.set("spark.driver.memory", "512m")

sparkConf.set("spark.executor.memory", "512m")

sparkConf.set("spark.kubernetes.pyspark.pythonVersion", "3")

sparkConf.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")

sparkConf.set("spark.kubernetes.authenticate.serviceAccountName", "spark")

sparkConf.set("spark.driver.port", "29415")

sparkConf.set("spark.driver.bindAddress", "0.0.0.0")

sparkConf.set("spark.driver.host", "52.153.234.86") #external IP address which AKS can connect to.

# Initialize our Spark cluster, this will actually

# generate the worker nodes.

spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()

sc = spark.sparkContext import osfrom pyspark import SparkContext, SparkConffrom pyspark.sql import SparkSession# Create Spark config for our Kubernetes based cluster managersparkConf = SparkConf()sparkConf.setMaster("k8s:// https://<AKS-API-domain>:443 ")sparkConf.setAppName("spark")sparkConf.set("spark.kubernetes.container.image", "tarrantro/spark-py:v3.0.1") #change that to your pyspark imagesparkConf.set("spark.kubernetes.namespace", "spark")sparkConf.set("spark.executor.instances", "3")sparkConf.set("spark.executor.cores", "2")sparkConf.set("spark.driver.memory", "512m")sparkConf.set("spark.executor.memory", "512m")sparkConf.set("spark.kubernetes.pyspark.pythonVersion", "3")sparkConf.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")sparkConf.set("spark.kubernetes.authenticate.serviceAccountName", "spark")sparkConf.set("spark.driver.port", "29415")sparkConf.set("spark.driver.bindAddress", "0.0.0.0")sparkConf.set("spark.driver.host", "52.153.234.86") #external IP address which AKS can connect to.# Initialize our Spark cluster, this will actually# generate the worker nodes.spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()sc = spark.sparkContext

Please notice, VM to listen on the setting of spark.driver.bindAddress. You may configure it as your internal address as long as AKS can access to it. So as to the setting of spark.driver.host. That is the IP address which executors in AKS will connect to. In my case, I use external IP of my VM.

spark.driver.port is an unused port which driver will open for listening. You can pick anyone you want.

Please make sure your VM can connect to the AKS API, as well as the AKS can connect to the VM.

You should also white list the port range 30000–60000 from AKS to VM in your network secutiry group.

Below is the result.

tarrant@testubuntujp1:~$ kubectl get pods -n spark

NAME READY STATUS RESTARTS AGE

my-notebook-deployment-7fb5cfbcf7-4kb98 1/1 Running 0 8d

spark-b5841a76505a06db-exec-1 1/1 Running 0 6m40s

spark-b5841a76505a06db-exec-2 1/1 Running 0 6m40s

spark-b5841a76505a06db-exec-3 1/1 Running 0 6m39s

test 1/1 Running 0 9d

You may use the code in method 1 to test.

Extinguish the Spark

After you have completed your development, you can shutdown your Spark cluster by simply executing:

sc.stop()

That will instruct your SparkContext to tear down the worker Pods you created above via the kube-apiserver.

To verify:

$ kubectl get pods -n spark

NAME READY STATUS RESTARTS AGE

my-notebook-deployment-7bf574447c-pdn2q 1/1 Running 0 20m

spark-1581043649282-exec-1 0/1 Terminating 0 16m

spark-1581043651262-exec-2 1/1 Terminating 0 16m

spark-1581043651392-exec-3 1/1 Terminating 0 16m

spark-1581043651510-exec-4 0/1 Terminating 0 16m

spark-1581043651623-exec-5 0/1 Terminating 0 16m

spark-1581043656753-exec-6 0/1 Terminating 0 16m

spark-1581043656875-exec-7 0/1 Terminating 0 16m

And now you are left with just your deployment/driver Pod in the spark namespace.