How to run a Spark application from an EC2 Instance
By: Natalie Olivo
Tags: Spark, Ec2, AWS, Hadoop, Git

Why would you do this instead of using EMR? Well, great question. Under certain circumstances, using EC2 might be cheaper than using EMR, but otherwise, EMR may be advisable. In any case, here’s how to run a Spark application from an EC2 instance:

Photo by Steve Richey on Unsplash

I used a Deep Learning AMI (Ubuntu 16.04) Version 25.3 with a p3 instance, for accelerated computing.

SSH into your EC2 instance.

ssh -i pem_key.pem ubuntu@public_dns_key

From the beginning, you have a few things installed.

You type into your EC2 terminal:

java -version

it returns:

openjdk version “1.8.0_222” OpenJDK Runtime Environment (build 1.8.0_222–8u222-b10–1ubuntu1~16.04.1-b10) OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

Java 8 is what we want for Spark to run, so this is good.

My application is written using python so I want to check that it’s installed.

python --version

It returns:

Python 3.6.6 :: Anaconda, Inc.

Great!

Now you’ll need to install Hadoop.

I used the following guidelines:

https://datawookie.netlify.com/blog/2017/07/installing-hadoop-on-ubuntu/

Go to the Spark downloads site and see which version of Hadoop it uses:

It uses Hadoop 2.7, as of November 2019, this may be different for you.

2. It uses Hadoop 2.7. Ok now go to the Hadoop mirror site and use wget

Right click, copy link to the Hadoop-2.7.7.tar.gz

type into your ubuntu terminal (paste what you just copied, I bolded it to let you know that yours may be different):

3. Unpack the zipped contents

tar -xvf hadoop-2.7.7.tar.gz

4. Find the location of java

type -p javac|xargs readlink -f|xargs dirname|xargs dirname

It returns:

/usr/lib/jvm/java-8-openjdk-amd64

Ok copy your output ^

5. Ok now edit the Hadoop configuration file so it can interact with java

vi hadoop-2.7.7/etc/hadoop/hadoop-env.sh

Type i to insert and update the JAVA_HOME variable by pasting that output you copied

To exit vim, use ESC + :wq! (wq stands for write and quit, the explanation point is to force it)

6. Set HADOOP_HOME and JAVA_HOME environment variables. The JAVA_HOME environment variable points to the directory where the Java runtime environment (JRE) is installed on your computer. The purpose is to point to where Java is installed.

You can do this by using

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export HADOOP_HOME=/home/ubuntu/hadoop-2.7.7

and update your path:

export PATH=$PATH:$HADOOP_HOME/bin/

But this might not save if you shut down your EC2 instance. So I do it by using vim and adding each of these exports to my .bashrc file.

vi .bashrc , i for insert, then :wq when you’re done.

PS: this is how you remove duplicates in your PATH, you don’t need to, they don’t hurt anything, but in case you were wondering:

if [ -n "$PATH" ]; then

old_PATH=$PATH:; PATH=

while [ -n "$old_PATH" ]; do

x=${old_PATH%%:*} # the first remaining entry

case $PATH: in

*:"$x":*) ;; # already there

*) PATH=$PATH:$x;; # not there yet

esac

old_PATH=${old_PATH#*:}

done

PATH=${PATH#:}

unset old_PATH x

fi

Source: https://unix.stackexchange.com/questions/40749/remove-duplicate-path-entries-with-awk-command

7. Source your .bashrc file, in your home directory type source .bashrc

8. Now to check version, you can go back to your home directory and type

hadoop version, and it should tell you

9. You can remove your zipped tar file from your home directory: rm hadoop-2.7.7.tar.gz

Now you’ll need to install Spark. I followed these guidelines:

https://datawookie.netlify.com/blog/2017/07/installing-spark-on-ubuntu/

1. Go to this site and copy link address of the first mirror site

2. In your ubuntu terminal type wget and paste the copied link

3. Now unpack the zipped tar file

tar -xvf spark-2.4.4-bin-hadoop2.7.tgz

4. Add spark to your environment variables export SPARK_HOME=/home/ubuntu/spark-2.4.4-bin-hadoop2.7

5. Install scala. Why do we do it the easy way here? Because we won’t need to peer into the jar files or anything.

sudo apt install scala

Now you have to make sure Spark, Hadoop, and Java can all work together. In our case, this also includes being able to read and write from S3.

1. Configure Spark

https://stackoverflow.com/questions/58415928/spark-s3-error-java-lang-classnotfoundexception-class-org-apache-hadoop-f

a. Navigate to folder ~/spark-2.4.4-bin-hadoop2.7/conf in your EC2 terminal

Run code:

touch spark_defaults.conf vi spark_defaults.conf

In here you’re going to want to add the following lines:

If 2FA applies to you: make sure your access key and secret key are for a service account (not associated with a username/password)

spark.hadoop.fs.s3a.access.key ***your_access_key*** spark.hadoop.fs.s3a.secret.key ***your_secret_key*** spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem spark.driver.extraClassPath /home/ubuntu/spark-2.4.4-bin-hadoop2.7/jars/hadoop-aws-2.7.3.jar:/home/ubuntu/spark-2.4.4-bin-hadoop2.7/jars/aws-java-sdk-1.7.4.jar

b. How do you make sure you have these jar files right?? I bolded them because they may be different for you. So to check, go to your Hadoop jars.

Go to the following folder using cd: cd ~/hadoop-2.7.7/share/hadoop/tools/lib and check to see which JARS are being used for aws-java-sdk and hadoop-aws , make sure these .jar files match what you just put in spark_defaults.conf .

c. Copy these files to the spark jars folder:

For the following code to work, be in the ~/hadoop-2.7.7/share/hadoop/tools/lib folder

cp hadoop-aws-2.7.3.jar ~/spark-2.4.4-bin-hadoop2.7/jars/ cp aws-java-sdk-1.7.4.jar ~/spark-2.4.4-bin-hadoop2.7/jars/

Configure Hadoop

a. Make it so each of your worker nodes can access S3 (since our code reads and writes from S3)

https://github.com/CoorpAcademy/docker-pyspark/issues/13

Edit file hadoop-2.7.7/etc/hadoop/core-site.xml to contain the following lines:

<configuration>

<property>

<name>fs.s3.awsAccessKeyId</name>

<value>*****</value>

</property> <property>

<name>fs.s3.awsSecretAccessKey</name>

<value>*****</value>

</property>

</configuration>

b. Copy the following jar files to the hadoop-2.7.7/share/hadoop/common/lib directory

sudo cp hadoop-2.7.7/share/hadoop/tools/lib/aws-java-sdk-1.7.4.jar hadoop-2.7.7/share/hadoop/common/lib/ sudo cp hadoop-2.7.7/share/hadoop/tools/lib/hadoop-aws-2.7.5.jar Hadoop-2.7.7/share/hadoop/common/lib/

Ok now you’re ready to clone your repository

git clone path/to/your/repo.git

Make sure to add a PYTHONPATH to this folder by adding the following line to your .bashrc file:

export PYTHONPATH=$PYTHONPATH:/home/ubuntu/repo

PYTHONPATH is an environment variable which you can set to add additional directories where python will look for modules and packages. For most installations, you should not set these variables since they are not needed for Python to run. Python knows where to find its standard library.

The only reason to set PYTHONPATH is to maintain directories of custom Python libraries that you do not want to install in the global default location (i.e., the site-packages directory).

Source: https://www.tutorialspoint.com/What-is-PYTHONPATH-environment-variable-in-Python

Now run your code!

Happy Sparking!

For more information on using Spark in EC2

check out Snipe.gg’s blog:

In their case, they found

[…] upon checking the EMR Pricing, we realized that EMR adds a pricing overhead of a whopping 25% to the price of the EC2 instances it uses

This blog also does a great job of highlighting tools and considerations for productionizing the effort I just outlined.