Spark Getting started-From source code to new SparkContext
Tags: Build, Apache Spark, Maven, Getting Started, Source Code

This post speaks about how to run Apache Spark on Mac(nix based systems) from Spark’s source code.

Though there are many “Spark Getting started” guides, there is none that get someone started from source code to writing a Spark application.

The Building Spark section of Open-source documentation is elaborate and talks about compiling and creating jars from Spark code. This post will help you run Spark using those jars, in Spark local mode.

Why do I need a local setup?

Few reasons that come to my mind

Quick iteration of your code changes — When you make a change to code, you want to see it in action. In a distributed system like Spark, unless you have a local setup, you will have to deploy your changes(scp!!!) to the cluster and verify. Why do you have to pay the cloud vendors just to verify a change that could as well be done locally? Increasing the debuggability of your application — 1. It is almost impossible to remotely debug a JVM running on a cloud VM.(Losing connection to you VM, round trip delays(~10s for a line of code!!!)). 2. Why not add log lines? Ok. Add a few log lines, build, deploy to the cluster. Now, you found that the log line is not at the right place. Re-iterate by adding new log lines(This implicitly translates to item 1). Had there been a local VM running the code, you could have attached a debugger to the JVM and gained insights into your Spark code.

Where is this setup useful?

If you want to get your hands dirty with Spark without an actual cluster. Such as

1. Getting started with Spark

2. Contributing code to Spark codebase.

Note: There are no actual executors(workers) in Spark local mode. Java threads act as executors. When you want to verify a piece of code change/operation that needs to run in a distributed manner, Spark local mode is not helpful as there is a single JVM which acts as both driver and executor.

Building Spark

Your Spark build will depend on answers to the following questions

Hadoop version(default — 3.2.0)? Hive support required? If yes, Hive version to use(default- 2.3.6)? Resource-manager(Standalone, YARN, mesos, K8S) — In the context of this post, Resource-manager is not relevant, since we will run Spark on Local mode.

Your build command will look like

./build/mvn -Phive -DskipTests clean package

(Note: this link has details on setting up maven. Also, a basic primer on maven will be helpful to connect pieces of the build process)

The command uses Hadoop version as 3.2.0 and Hive version as 2.3.6 which are the defaults.

Note: Add the following options to the build command, if required(In an ideal scenario, these options should never be required) 1. -Denforcer.skip - If you get an error message like "Some Enforcer rules have failed"

2. -Drat.skip - failures in license check

3. -Dscala.skip - To skip style checks

Effects of the firing the command

Let us assume that code has been downloaded at SPARK_HOME directory

$SPARK_HOME/assembly/target/scala-2.12/jars directory should have all the jars — Spark + Hadoop(default 3.2.0) + Hive(2.3.6) required for you to start an Spark application.

Launching spark-shell

$SPARK_HOME/bin/spark-shell --conf spark.master=local

Let us dissect the above command

$SPARK_HOME/bin/spark-shell- Shell script which fires up a scala shell with a pre-created SparkContext(bound to variable sc) and SparkSession(bound to variable spark) objects. — conf- allows you to pass configs to spark application. It should be followed by <key>=<value>. spark.master=local- Indicates that Spark application should start in local mode, where the driver and executors(Java threads, in this case) run in single JVM.

If the command runs successfully, you should see

This should generally work out of the box. Why? Since all the Hadoop jars , Hive jars required for Spark is made available in assembly/target.