Spark installation on single node
By: Sandeep Singh
Tags: Big Data, Data Engineering, Data Science, Apache Spark

Photo by Jez Timms on Unsplash

In addition to running the spark on the YARN or Mesos cluster managers, Spark also provides a simple standalone deploy mode. You can set up and launch a standalone cluster or set up on a single machine for the personal development or testing purpose. In this post I will be providing steps to install and configure spark on single node.

Steps to Apache Spark Installation in Standalone Mode on single machine.

before proceeding further, make sure you have compatible java installed on your machine. You can verify it by typing command:

[user@hostname ~]$ java -version

java version "1.8.0_112"

Java(TM) SE Runtime Environment (build 1.8.0_112-b15)

Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

[user@hostname ~]$

For this installation, I will be using Spark 2.3.4 which requires Java 8

Now download the compiled version of Spark on some directory of the machine.

Extract the tar file

tar -zxvf spark-2.3.4-bin-hadoop2.7.tgz

where,

x for extract

v for verbose

z for gnuzip

f for file, should come at last just before file name.

Set the Spark classpath in your environment

Place below lines in your file.

export SPARK_HOME=/home/user/spark-2.3.4-bin-hadoop2.7

export PATH=$SPARK_HOME/bin:$PATH [user@hostname ~]$ source ~/.bashrc

Now start master

[user@hostname spark-2.3.4-bin-hadoop2.7]$ ./sbin/start-master.sh

Inspect the response:

user@hostname spark-2.3.4-bin-hadoop2.7]$ ./sbin/start-master.sh

starting org.apache.spark.deploy.master.Master, logging to /home/user/spark-2.3.4-bin-hadoop2.7/logs/spark-user-org.apache.spark.deploy.master.Master-1-hostname.out

[user@hostname spark-2.3.4-bin-hadoop2.7]$

Now inspect the .out file



2019-09-12 13:34:45 INFO SignalUtils:54 - Registered signal handler for TERM

2019-09-12 13:34:45 INFO SignalUtils:54 - Registered signal handler for HUP

2019-09-12 13:34:45 INFO SignalUtils:54 - Registered signal handler for INT

2019-09-12 13:34:46 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2019-09-12 13:34:46 INFO SecurityManager:54 - Changing view acls to: user

2019-09-12 13:34:46 INFO SecurityManager:54 - Changing modify acls to: user

2019-09-12 13:34:46 INFO SecurityManager:54 - Changing view acls groups to:

2019-09-12 13:34:46 INFO SecurityManager:54 - Changing modify acls groups to:

2019-09-12 13:34:46 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set()

2019-09-12 13:34:46 INFO Utils:54 - Successfully started service 'sparkMaster' on port 7077.

2019-09-12 13:34:46 INFO Master:54 - Starting Spark master at spark://hostname.com:7077

2019-09-12 13:34:46 INFO Master:54 - Running Spark version 2.3.4

2019-09-12 13:34:46 INFO log:192 - Logging initialized @1352ms

2019-09-12 13:34:46 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown

2019-09-12 13:34:46 INFO Server:419 - Started @1414ms

2019-09-12 13:34:46 INFO AbstractConnector:278 - Started ServerConnector@f3dc25a{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}

2019-09-12 13:34:46 INFO Utils:54 - Successfully started service 'MasterUI' on port 8080.

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5a77913e{/app,null,AVAILABLE,@Spark}

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51d28d9d{/app/json,null,AVAILABLE,@Spark}

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5b3b13c0{/,null,AVAILABLE,@Spark}

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@324cc177{/json,null,AVAILABLE,@Spark}

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77370cc6{/static,null,AVAILABLE,@Spark}

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26ec989{/app/kill,null,AVAILABLE,@Spark}

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e84e949{/driver/kill,null,AVAILABLE,@Spark}

2019-09-12 13:34:46 INFO MasterWebUI:54 - Bound MasterWebUI to 0.0.0.0, and started at

2019-09-12 13:34:46 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6fb6cb3f{/,null,AVAILABLE}

2019-09-12 13:34:46 INFO AbstractConnector:278 - Started ServerConnector@46a58984{HTTP/1.1,[http/1.1]}{hostname.com:6066}

2019-09-12 13:34:46 INFO Server:419 - Started @1532ms

2019-09-12 13:34:46 INFO Utils:54 - Successfully started service on port 6066.

2019-09-12 13:34:46 INFO StandaloneRestServer:54 - Started REST server for submitting applications on port 6066

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6384c013{/metrics/master/json,null,AVAILABLE,@Spark}

2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3d2ad11b{/metrics/applications/json,null,AVAILABLE,@Spark}

2019-09-12 13:34:47 INFO Master:54 - I have been elected leader! New state: ALIVE

2019-09-12 13:41:09 INFO Master:54 - Registering worker 199.6.212.152:35633 with 32 cores, 124.6 GB RAM 2019-09-12 13:34:45 INFO Master:2612 - Started daemon with process name: 143038@hostname2019-09-12 13:34:45 INFO SignalUtils:54 - Registered signal handler for TERM2019-09-12 13:34:45 INFO SignalUtils:54 - Registered signal handler for HUP2019-09-12 13:34:45 INFO SignalUtils:54 - Registered signal handler for INT2019-09-12 13:34:46 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2019-09-12 13:34:46 INFO SecurityManager:54 - Changing view acls to: user2019-09-12 13:34:46 INFO SecurityManager:54 - Changing modify acls to: user2019-09-12 13:34:46 INFO SecurityManager:54 - Changing view acls groups to:2019-09-12 13:34:46 INFO SecurityManager:54 - Changing modify acls groups to:2019-09-12 13:34:46 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set()2019-09-12 13:34:46 INFO Utils:54 - Successfully started service 'sparkMaster' on port 7077.2019-09-12 13:34:46 INFO Master:54 - Starting Spark master at spark://hostname.com:70772019-09-12 13:34:46 INFO Master:54 - Running Spark version 2.3.42019-09-12 13:34:46 INFO log:192 - Logging initialized @1352ms2019-09-12 13:34:46 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown2019-09-12 13:34:46 INFO Server:419 - Started @1414ms2019-09-12 13:34:46 INFO AbstractConnector:278 - Started ServerConnector@f3dc25a{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}2019-09-12 13:34:46 INFO Utils:54 - Successfully started service 'MasterUI' on port 8080.2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5a77913e{/app,null,AVAILABLE,@Spark}2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51d28d9d{/app/json,null,AVAILABLE,@Spark}2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5b3b13c0{/,null,AVAILABLE,@Spark}2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@324cc177{/json,null,AVAILABLE,@Spark}2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77370cc6{/static,null,AVAILABLE,@Spark}2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26ec989{/app/kill,null,AVAILABLE,@Spark}2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e84e949{/driver/kill,null,AVAILABLE,@Spark}2019-09-12 13:34:46 INFO MasterWebUI:54 - Bound MasterWebUI to 0.0.0.0, and started at http://hostname.com:8080 2019-09-12 13:34:46 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6fb6cb3f{/,null,AVAILABLE}2019-09-12 13:34:46 INFO AbstractConnector:278 - Started ServerConnector@46a58984{HTTP/1.1,[http/1.1]}{hostname.com:6066}2019-09-12 13:34:46 INFO Server:419 - Started @1532ms2019-09-12 13:34:46 INFO Utils:54 - Successfully started service on port 6066.2019-09-12 13:34:46 INFO StandaloneRestServer:54 - Started REST server for submitting applications on port 60662019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6384c013{/metrics/master/json,null,AVAILABLE,@Spark}2019-09-12 13:34:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3d2ad11b{/metrics/applications/json,null,AVAILABLE,@Spark}2019-09-12 13:34:47 INFO Master:54 - I have been elected leader! New state: ALIVE2019-09-12 13:41:09 INFO Master:54 - Registering worker 199.6.212.152:35633 with 32 cores, 124.6 GB RAM

Start Worker and register the worker with master

vi the .out file which you get after start master and copy the master url

spark://hostname.com:7077

now start the worker and register it with master using following command

./sbin/start-slave.sh spark://hostname.com:7077

Response:

[user@hostname spark-2.3.4-bin-hadoop2.7]$ vi /home/user/spark-2.3.4-bin-hadoop2.7/logs/spark-user-org.apache.spark.deploy.master.Master-1-hostname.out

[user@hostname spark-2.3.4-bin-hadoop2.7]$ ./sbin/start-slave.sh spark://hostname.com:7077

starting org.apache.spark.deploy.worker.Worker, logging to /home/user/spark-2.3.4-bin-hadoop2.7/logs/spark-user-org.apache.spark.deploy.worker.Worker-1-hostname.out

[user@hostname spark-2.3.4-bin-hadoop2.7]$

Now inspect the .out file, you will see the log like this



2019-09-12 13:41:07 INFO SignalUtils:54 - Registered signal handler for TERM

2019-09-12 13:41:07 INFO SignalUtils:54 - Registered signal handler for HUP

2019-09-12 13:41:07 INFO SignalUtils:54 - Registered signal handler for INT

2019-09-12 13:41:08 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2019-09-12 13:41:08 INFO SecurityManager:54 - Changing view acls to: user

2019-09-12 13:41:08 INFO SecurityManager:54 - Changing modify acls to: user

2019-09-12 13:41:08 INFO SecurityManager:54 - Changing view acls groups to:

2019-09-12 13:41:08 INFO SecurityManager:54 - Changing modify acls groups to:

2019-09-12 13:41:08 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set()

2019-09-12 13:41:08 INFO Utils:54 - Successfully started service 'sparkWorker' on port 35633.

2019-09-12 13:41:08 INFO Worker:54 - Starting Spark worker 100.2.101.101:35633 with 32 cores, 124.6 GB RAM

2019-09-12 13:41:08 INFO Worker:54 - Running Spark version 2.3.4

2019-09-12 13:41:08 INFO Worker:54 - Spark home: /home/user/spark-2.3.4-bin-hadoop2.7

2019-09-12 13:41:08 INFO log:192 - Logging initialized @1510ms

2019-09-12 13:41:08 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown

2019-09-12 13:41:08 INFO Server:419 - Started @1576ms

2019-09-12 13:41:08 INFO AbstractConnector:278 - Started ServerConnector@3f9e3902{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}

2019-09-12 13:41:08 INFO Utils:54 - Successfully started service 'WorkerUI' on port 8081.

2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1dc21140{/logPage,null,AVAILABLE,@Spark}

2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5896ed4f{/logPage/json,null,AVAILABLE,@Spark}

2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d9a25f0{/,null,AVAILABLE,@Spark}

2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1ad57f24{/json,null,AVAILABLE,@Spark}

2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@754605a4{/static,null,AVAILABLE,@Spark}

2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c5e9251{/log,null,AVAILABLE,@Spark}

2019-09-12 13:41:08 INFO WorkerWebUI:54 - Bound WorkerWebUI to 0.0.0.0, and started at

2019-09-12 13:41:08 INFO Worker:54 - Connecting to master hostname.com:7077...

2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4ac9255f{/metrics/json,null,AVAILABLE,@Spark}

2019-09-12 13:41:08 INFO TransportClientFactory:267 - Successfully created connection to hostname.com/199.6.212.152:7077 after 40 ms (0 ms spent in bootstraps)

2019-09-12 13:41:09 INFO Worker:54 - Successfully registered with master spark://hostname.com:7077 2019-09-12 13:41:07 INFO Worker:2612 - Started daemon with process name: 144697@hostname2019-09-12 13:41:07 INFO SignalUtils:54 - Registered signal handler for TERM2019-09-12 13:41:07 INFO SignalUtils:54 - Registered signal handler for HUP2019-09-12 13:41:07 INFO SignalUtils:54 - Registered signal handler for INT2019-09-12 13:41:08 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2019-09-12 13:41:08 INFO SecurityManager:54 - Changing view acls to: user2019-09-12 13:41:08 INFO SecurityManager:54 - Changing modify acls to: user2019-09-12 13:41:08 INFO SecurityManager:54 - Changing view acls groups to:2019-09-12 13:41:08 INFO SecurityManager:54 - Changing modify acls groups to:2019-09-12 13:41:08 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set()2019-09-12 13:41:08 INFO Utils:54 - Successfully started service 'sparkWorker' on port 35633.2019-09-12 13:41:08 INFO Worker:54 - Starting Spark worker 100.2.101.101:35633 with 32 cores, 124.6 GB RAM2019-09-12 13:41:08 INFO Worker:54 - Running Spark version 2.3.42019-09-12 13:41:08 INFO Worker:54 - Spark home: /home/user/spark-2.3.4-bin-hadoop2.72019-09-12 13:41:08 INFO log:192 - Logging initialized @1510ms2019-09-12 13:41:08 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown2019-09-12 13:41:08 INFO Server:419 - Started @1576ms2019-09-12 13:41:08 INFO AbstractConnector:278 - Started ServerConnector@3f9e3902{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}2019-09-12 13:41:08 INFO Utils:54 - Successfully started service 'WorkerUI' on port 8081.2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1dc21140{/logPage,null,AVAILABLE,@Spark}2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5896ed4f{/logPage/json,null,AVAILABLE,@Spark}2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d9a25f0{/,null,AVAILABLE,@Spark}2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1ad57f24{/json,null,AVAILABLE,@Spark}2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@754605a4{/static,null,AVAILABLE,@Spark}2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c5e9251{/log,null,AVAILABLE,@Spark}2019-09-12 13:41:08 INFO WorkerWebUI:54 - Bound WorkerWebUI to 0.0.0.0, and started at http://hostname.com:8081 2019-09-12 13:41:08 INFO Worker:54 - Connecting to master hostname.com:7077...2019-09-12 13:41:08 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4ac9255f{/metrics/json,null,AVAILABLE,@Spark}2019-09-12 13:41:08 INFO TransportClientFactory:267 - Successfully created connection to hostname.com/199.6.212.152:7077 after 40 ms (0 ms spent in bootstraps)2019-09-12 13:41:09 INFO Worker:54 - Successfully registered with master spark://hostname.com:7077

Your standalone cluster on single node is read now, you can test it using running command spark-shell



2019-09-12 13:30:45 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

Spark context Web UI available at

Spark context available as 'sc' (master = local[*], app id = local-1568287851585).

Spark session available as 'spark'. [user@hostname ~]$ spark-shell2019-09-12 13:30:45 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicableSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).Spark context Web UI available at http://hostname.com:4040 Spark context available as 'sc' (master = local[*], app id = local-1568287851585).Spark session available as 'spark'.

You can exit the spark shell using typing :q then enter

Now submit example application using spark-submit

spark-submit -class org.apache.spark.examples.SparkPi -master local[8] /home/user/spark-2.3.4-bin-hadoop2.7/examples/jars/spark-examples_2.11–2.3.4.jar

[user@hostname jars]$ spark-submit --class org.apache.spark.examples.SparkPi --master local[8] /home/user/spark-2.3.4-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.4.jar

If this example execute successfully, your spark installation is fine. you can see the results in console log

2019-09-12 13:53:27 INFO DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 0.615754 s

Pi is roughly 3.1416557082785412

2019-09-12 13:53:27 INFO AbstractConnector:318 - Stopped Spark@6914bc2c{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

Now you can submit your own jar

[user@hostname ~]$ spark-submit --class com.spark.example.DataValidation --master local[2] /home/user/spark-dq/dataquality-framework-0.0.1.jar

You are done with the spark standalone cluster setup on single node. If you see any issue here then you need to inspect your code.

Happy learning, if you liked my post please don’t hesitate to clap and follow me.

Thanks for the reading my blog. If you have any specific questions where I might help you out, don’t hesitate to reach out to me!