How to build Spark from source and deploy it to a Kubernetes cluster in 60 minutes | by Nikolay Dimolarov | Towards Data Science

Motivation/Prelude
In my last article I explained the broad strokes of the Hadoop ecosystem and you can read about it here. What is important about that article is the end, which I will blatantly plagiarise from my other article as it serves as the beginning of this one too:
Now, if you have been sort of listening in on Hadoop’s ecosystem over the last few years you would have seen that the two biggest players on the market — Cloudera and Hortonworks — merged around a year ago amid a slow down of the Hadoop big data market. Add the fact that people seem way more interested in Kubernetes than in older Hadoop specific technologies like YARN for resource management and orchestration, the fast adoption of DL frameworks like PyTorch and you sort of have a perfect storm forming for the ageing Hadoop stack. Nonetheless projects like Apache Spark are chugging along by e.g. introducing Kubernetes as an alternative to YARN. Exciting times for the ecosystem!
Introduction
The goal of this article is to show you what some of the cool kids are doing in 2020 in the Big Data ecosystem; which is trying to cram stuff into Kubernetes (which is a good thing!). More specifically using Spark’s experimental implementation of a native Spark Driver and Executor where Kubernetes is the resource manager (instead of e.g. YARN)
... and let us do this in 60 minutes:
Clone Spark project from GitHubBuild Spark distribution with MavenBuild Docker Image locallyRun Spark Pi job with multiple executor replicasUse port forwarding to show the Spark UI in your browser and inspect your Spark job
Clone Spark project from GitHub
Build Spark distribution with Maven
Build Docker Image locally
Run Spark Pi job with multiple executor replicas
Use port forwarding to show the Spark UI in your browser and inspect your Spark job
If it is that simple why do we need this article?! Read along to find out how this took me days to figure out the first time around.
Disclaimer: your mileage on the 60 minutes might vary but it is really doable under the assumption that you generally know how do the things on a computer including setting up a local k8s cluster and running bash scripts and the likes. Furthermore, building Spark might take a while if you have a slowish computer ;)
Now that everyone is on board, let’s deploy Spark on Kubernetes. For this you can use your laptop’s run of the mill minikube setup instead of renting a server in a public cloud just for this exercise. Unless you want to go all in on this, in which case you have just been saluted.
Steps 1–3 (clone repo, build Spark, build Docker image):
This is actually where the fun starts — on the “easiest” steps. Well, put on your seatbelt and check this out (pun intended):
If you clone the official Spark repository here and innocently follow the official Spark guide on running Spark in k8s here, you will run into this issue that I opened in Spark’s Jira backlog a few days ago.
Namely, that there are multiple wrong references in the Dockerfile and because of that simply running a docker build command as described in the Dockerfile comments will not work.
Update: Well, it turns out that you can in fact run things as described in the documentation but only if you pay really close attention.
Instead of running
./build/mvn -Pkubernetes -DskipTests clean package
you need to run
dev/make-distribution.sh -Pkubernetes
which creates a Spark distribution instead of just the normal assembly bits and pieces but I guess I skipped on the fine-print in their tutorial, so I updated this article accordingly.
tl;dr to complete steps 1–3 simply do this:
git clone git@github.com:apache/spark.gitcd sparkdev/make-distribution.sh -Pkubernetescd distdocker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .
At this point you should have an image of Spark in your local Docker registry!
Step 4: Run Spark Pi job with multiple executor replicas in Kubernetes:
What the Spark article I linked above mentions but does not quite explain as a showstopper is the fact that due to Kubernetes’ RBAC (role-based access control), you cannot simply deploy Spark into your cluster as Spark needs some additional rights to your Kubernetes cluster to manage pods. This is due to Spark’s architecture — you deploy a Spark Driver, which can then create the Spark Executors in pods and then clean them up once the job is done:
tl;dr we need to create a service account with kubectl for Spark:
kubectl create serviceaccount sparkkubectl create clusterrolebinding spark-role --clusterrole=edit  --serviceaccount=default:spark --namespace=default
Next up is to run Spark Pi with our locally built Docker image:
bin/spark-submit \--master k8s://https://kubernetes.docker.internal:6443 \--deploy-mode cluster \--name spark-pi \--class org.apache.spark.examples.SparkPi \--conf spark.executor.instances=2 \--conf spark.kubernetes.container.image=spark:latest \--conf spark.kubernetes.container.image.pullPolicy=Never \--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \local:///opt/spark/examples/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar 10000000
Ok, but what is actually happening here. Well, this is how you deploy your Spark Driver into your Kubernetes cluster! Let us go through the parameters, so you can actually start working with this on your own afterwards:
define Kubernetes cluster (find with kubectl cluster-info). Yes, the somewhat weird k8s:// prefix is needed.
define deployment mode (cluster, duh)
define name for Spark Driver (that’s also what your pod’s name will start with)
define Spark Pi example
run the Spark Executor with 2 replicas on Kubernetes that will be spawned by your Spark Driver
use our local spark:latest image
define the Kubernetes image pull policy to Never, so the local image with that name can be used. If one is not very familiar with the inner workings of k8s, this could definitely take a minute to figure out...
define service account (remember RBAC?)
point to local jar path (which is copied with all other examples in the Dockerfile to the specified path) with the argument 10000000 (if you do not know what that number is for, examine the Spark Pi source code and documentation) . Yes, the local:/// is correct and it’s not a typo.
kubectl get pods
Should now return a list of the running pods! And do not worry when they get terminated in the end — that is the default setting for this implementation.
Step 5: Use port forwarding to show Spark UI
kubectl port-forward <insert spark driver pod name> 4040:4040
Then you should be able to access the Spark UI with the localhost:4040 from the first command from above in your browser like so:
You can also check out your logs like this:
kubectl -n=default logs -f <insert spark driver pod name>
Conclusion
Doing this exact setup with the config and the Dockerfile can be either super straightforward if you know Spark and Kubernetes really well or a little bit of a nightmare if you do not. I hope that this will help you to do this in minutes instead!
Where do you go from here? Anywhere you want. The goal of this article was to quickly get you up and running with a new fancy resource manager for Spark. I suggest you play around with this setup with other Spark apps as the next steps — I might just write an article on some more complicated examples in the future (let me know if I should).
Have fun with Kubernetes!
Alternatives
This is an update from 20th April 2020 — one can also use Google’s native Kubernetes operator, which seems very promising and can remove the manual deployment steps into your cluster:
github.com
It is currently being used by e.g. Salesforce and Microsoft in production and is being evaluated for production by Uber and Lyft. Something to look out for in the future!