Spark on Kubernetes
By: Scott Haines
Tags: Docker, Engineering, Apache Spark, Kubernetes, DevOps

Spark on Kubernetes: Hello World

We are now preparing to run our first Spark Job on Kubernetes. This is a big deal. This is why we are here!

We need to find the cluster master.

kubectl cluster-info

Remember This: Now we are actually using the values from the Cluster Info

What we care about in our case is the Kubernetes Master since we need to reference that in our Spark Submit.

$SPARK_HOME/bin/spark-submit \

--master k8s://https://kubernetes.docker.internal:6443 \

--deploy-mode cluster \

--name spark-pi \

--class org.apache.spark.examples.SparkPi \

--conf spark.executor.instances=5 \

--conf spark.kubernetes.container.image=newfrontdocker/spark:v3.0.1-j14 \

local:///opt/spark/examples/jars/spark-examples_2.12–3.0.1.jar

It is worth pointing out that the local:///opt/ is actually referencing the JAR Path inside of the Docker Container. This can be useful when you want to access local configuration files, spark-environment variables, logging profiles, and so much more from within your deployment.

Spark Submit Output: You should see something similar when running your Spark application

Spark in a Pod

If you open up the Kubernetes Dashboard while your Spark Application is running you will see a similar view. It is worth noting that since we configured spark.executor.instances=5 in our Spark Submit that you will see 5 pods running for your application. This is equivalent to having 5 executor instances in traditional Spark Standalone world.

Kubernetes Dashboard: View of the Spark Executor Pods Running

That’s a Wrap

We really covered a lot of ground during this first tutorial. I really enjoy this part of the learning cycle. It is when the creative juices start flowing and new ideas seems to come alive, it is one of the brains ways of rewarding the learning process and I hope you were able to stick around up until this point. Over the course of this tutorial we covered a lot of ground fairly quickly. So here is a rewind.

Docker Desktop

We installed and got Docker Desktop up and Running locally on our computers. High Fives all around. This included provisioning some space on our laptops in order to power the Docker environment. I went with a 4 core by 10gb ram slice off the old laptop. If you have more cores and more ram, Kubernetes will gladly take that from you to power things to run quicker with more parallelism.

Local Kubernetes

We took things further with the click of a checkbox and enabled Kubernetes (single-node) cluster support in our local Docker. This was the start of our kubernetes journey together.

Kubernetes Dashboard

We fetched and applied the configuration and information for running the Kubernetes Dashboard so we could get some Eyes on the cluster. This is a much better view of the cluster than just having the Running green light on the Docker Desktop Dashboard.

We explored some of the `kubectl` commands to find our tokens and actually login into the dashboard

We saw how to get some really important cluster information by using the `kubectl cluster-info` command

Installing Apache Spark

We installed Apache Spark locally in order to gain access to the core Dockerfile that ships alongside the project. This is an important part of the process since when we create a new Docker image we need to be able to ship the Spark source locally as well.

If we were just looking to do some basic things and get to know Spark with the spark-shell then we could have used that out of the box to just play around with the Spark Runtime. $SPARK_HOME/bin/spark-shell and $SPARK_HOME/bin/pyspark respectively give you access to the local shell.

Building the Spark Docker Image

We were able to build a local version of Spark bringing our own JDK to the table openjdk:14-slim — cause diets are nice when shipping containers. This allowed us to choose how we want to build these images without bringing in the collective kitchen sink. The ability to test locally, list local images, remove images selectively and lastly pushing an image to a public repository is all in your toolbox now.

Running Spark Locally on Kubernetes

We finished up this whirlwind deep dive by running one of the example Spark applications. This was done as a HelloWorld for testing the cluster and now we can use the work completed here in order to move on to bigger and better things.

Next Steps

In my next tutorial I am going to show you how to use these two base Spark Images in order to simplify Running Spark Scala applications as well as PySpark applications that require custom python installations. We will look more into how Kubernetes configurations work in order to enable service and role based access to the processes running within our driver and executor pods.

Thanks for taking the time to dive into Spark on Kubernetes.