K8S Operating and Apache Spark
By: Tyler M.
Tags: Computer Science, Kubernetes, Computers, Apache Spark

Many of us have been thinking about different ways to submit files and applications to the cloud and cloud clusters. If you are running Kubernetes on the cloud — an open-source program that can be used for managing cloud clusters & automation/deployment of applications on the cloud — you might need a program to remotely send apps and files to Kubernetes & there is a program for that. This program is a suite of tools for running various Spark jobs on Kubernetes that is known as Kubernetes Operator for Spark. It can be implemented based around the typical methods that people use to put in place regular Kubernetes operator patterns. It supports mounting volumes of data and files called ConfigMaps that allow you to customize Kubernetes pods. It also provides a useful command line interface to manage various attempts to send applications and files to Kubernetes. Now, before you run Kubernetes Operator for Spark, you have to have a program installed on your Linux device known as helm. Helm is a program that can be installed to Ubuntu or LinuxMint using the command $sudo snap install helm — classic. You can also use the program curl on other versions of Linux in order to install helm:

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3

$ chmod 700 get_helm.sh

$ ./get_helm.sh

Afterwards, you need to include a helm chart in order to start working with helm for spark on Kubernetes using to the command:

$ helm repo add incubator https://charts.helm.sh/incubator --force-update

Now, you need to create a type of file called a .yaml file. YAML is specifically a type of data serialization language that is used to create configuration files to give information to a variety of programming language. This language is specifically designed to make it easier for humans to interpret and aligns everything for data, not documents. This yaml file can be used to generate ‘namespaces’ that can be used for spark with Kubernetes. Here is an example of the kind of .yaml file you want to produce:

apiVersion: v1

kind: Namespace

metadata:

name: spark-apps

---

apiVersion: v1

kind: Namespace

metadata:

name: spark-operator

This generates two different namespaces — spark-apps and spark-operator — once you enter the command kubectl create -f <name of yaml file>.yaml , which creates new namespaces necessary for running the spark apps you want on your Kubernetes pods and your digital cloud. Next, you want to create a service account for Kubernetes using the command kubectl create custerrolebinding spark-role — clusterrole=edit — serviceaccount=<namespace you choose>:spark — namespace=<namespace you choose>. The service account is an account that gives you the ability to create the pods (or segments) for running apps that Kubernetes uses. Without it, you are not going to be able to run any type of Spark apps on Kubernetes. After that, you need to create a yaml file that configures everything you need to configure the pod you want to run your Spark app on or submit a file to. Here is an example provided by Google of a .yaml file for running a basic spark app on the K8s cloud:

apiVersion: “sparkoperator.k8s.io/v1beta2”

kind: SparkApplication

metadata:

name: spark-pi

namespace: default

spec:

type: Scala

mode: cluster

image: “gcr.io/spark-operator/spark:v3.1.1”

imagePullPolicy: Always

mainClass: org.apache.spark.examples.SparkPi

mainApplicationFile: “local:///opt/spark/examples/jars/spark-examples_2.12–3.1.1.jar”

sparkVersion: “3.1.1”

restartPolicy:

type: Never

volumes:

— name: “test-volume”

hostPath:

path: “/tmp”

type: Directory

driver:

cores: 1

coreLimit: “1200m”

memory: “512m”

labels:

version: 3.1.1

serviceAccount: spark

volumeMounts:

— name: “test-volume”

mountPath: “/tmp”

executor:

cores: 1

instances: 1

memory: “512m”

labels:

version: 3.1.1

volumeMounts:

— name: “test-volume”

mountPath: “/tmp”

If everything is successful, you can type kubectl get pods and see something like this:

NAME READY STATUS RESTARTS AGE

spark-pi-driver 0/1 Completed 0 48s

After it submits, you can type a command like this:

kubectl get sparkapplications spark-pi -o=yaml

You should get a message that contains a message like this:

...

status:

applicationState:

state: COMPLETED

...

If that all works out then congrats! You can now submit apps & files to Kubernetes cloud clusters using Kubernetes Operator on Spark!