How to run a Spark application from an EC2 Instance | by Natalie Olivo | Towards Data Science

Why would you do this instead of using EMR? Well, great question. Under certain circumstances, using EC2 might be cheaper than using EMR, but otherwise, EMR may be advisable. In any case, here’s how to run a Spark application from an EC2 instance:
I used a Deep Learning AMI (Ubuntu 16.04) Version 25.3 with a p3 instance, for accelerated computing.
ssh -i pem_key.pem ubuntu@public_dns_key
You type into your EC2 terminal:
java -version
it returns:
openjdk version “1.8.0_222”OpenJDK Runtime Environment (build 1.8.0_222–8u222-b10–1ubuntu1~16.04.1-b10)OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
Java 8 is what we want for Spark to run, so this is good.
My application is written using python so I want to check that it’s installed.
python --version
It returns:
Python 3.6.6 :: Anaconda, Inc.
Great!
I used the following guidelines:
https://datawookie.netlify.com/blog/2017/07/installing-hadoop-on-ubuntu/
Go to the Spark downloads site and see which version of Hadoop it uses:
Go to the Spark downloads site and see which version of Hadoop it uses:
2. It uses Hadoop 2.7. Ok now go to the Hadoop mirror site and use wget
Right click, copy link to the Hadoop-2.7.7.tar.gz
type into your ubuntu terminal (paste what you just copied, I bolded it to let you know that yours may be different):
wget http://apache.mirrors.ionfish.org/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz
3. Unpack the zipped contents
tar -xvf hadoop-2.7.7.tar.gz
4. Find the location of java
type -p javac|xargs readlink -f|xargs dirname|xargs dirname
It returns:
/usr/lib/jvm/java-8-openjdk-amd64
Ok copy your output ^
5. Ok now edit the Hadoop configuration file so it can interact with java
vi hadoop-2.7.7/etc/hadoop/hadoop-env.sh
Type i to insert and update the JAVA_HOME variable by pasting that output you copied
To exit vim, use ESC + :wq! (wq stands for write and quit, the explanation point is to force it)
6. Set HADOOP_HOME and JAVA_HOME environment variables. The JAVA_HOME environment variable points to the directory where the Java runtime environment (JRE) is installed on your computer. The purpose is to point to where Java is installed.
You can do this by using
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64export HADOOP_HOME=/home/ubuntu/hadoop-2.7.7
and update your path:
export PATH=$PATH:$HADOOP_HOME/bin/
But this might not save if you shut down your EC2 instance. So I do it by using vim and adding each of these exports to my .bashrc file.
vi .bashrc , i for insert, then :wq when you’re done.
PS: this is how you remove duplicates in your PATH, you don’t need to, they don’t hurt anything, but in case you were wondering:
if [ -n "$PATH" ]; then    old_PATH=$PATH:; PATH=    while [ -n "$old_PATH" ]; do        x=${old_PATH%%:*}       # the first remaining entry        case $PATH: in            *:"$x":*) ;;          # already there            *) PATH=$PATH:$x;;    # not there yet        esac        old_PATH=${old_PATH#*:}    done    PATH=${PATH#:}    unset old_PATH xfi
Source: https://unix.stackexchange.com/questions/40749/remove-duplicate-path-entries-with-awk-command
7. Source your .bashrc file, in your home directory type source .bashrc
8. Now to check version, you can go back to your home directory and type  hadoop version, and it should tell you
9. You can remove your zipped tar file from your home directory: rm hadoop-2.7.7.tar.gz
https://datawookie.netlify.com/blog/2017/07/installing-spark-on-ubuntu/
1. Go to this site and copy link address of the first mirror site
2. In your ubuntu terminal type wget and paste the copied link
wget http://mirrors.sonic.net/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
3. Now unpack the zipped tar file
tar -xvf spark-2.4.4-bin-hadoop2.7.tgz
4. Add spark to your environment variables export SPARK_HOME=/home/ubuntu/spark-2.4.4-bin-hadoop2.7
5. Install scala. Why do we do it the easy way here? Because we won’t need to peer into the jar files or anything.
sudo apt install scala
1. Configure Spark https://stackoverflow.com/questions/58415928/spark-s3-error-java-lang-classnotfoundexception-class-org-apache-hadoop-f
a. Navigate to folder ~/spark-2.4.4-bin-hadoop2.7/conf in your EC2 terminal
Run code:
touch spark_defaults.confvi spark_defaults.conf
In here you’re going to want to add the following lines: If 2FA applies to you: make sure your access key and secret key are for a service account (not associated with a username/password)
spark.hadoop.fs.s3a.access.key ***your_access_key***spark.hadoop.fs.s3a.secret.key ***your_secret_key***spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystemspark.driver.extraClassPath /home/ubuntu/spark-2.4.4-bin-hadoop2.7/jars/hadoop-aws-2.7.3.jar:/home/ubuntu/spark-2.4.4-bin-hadoop2.7/jars/aws-java-sdk-1.7.4.jar
b. How do you make sure you have these jar files right?? I bolded them because they may be different for you. So to check, go to your Hadoop jars.
Go to the following folder using cd:cd ~/hadoop-2.7.7/share/hadoop/tools/liband check to see which JARS are being used for aws-java-sdk and hadoop-aws, make sure these .jar files match what you just put in spark_defaults.conf.
c. Copy these files to the spark jars folder:
For the following code to work, be in the ~/hadoop-2.7.7/share/hadoop/tools/lib folder
cp hadoop-aws-2.7.3.jar ~/spark-2.4.4-bin-hadoop2.7/jars/cp aws-java-sdk-1.7.4.jar ~/spark-2.4.4-bin-hadoop2.7/jars/
a. Make it so each of your worker nodes can access S3 (since our code reads and writes from S3) https://github.com/CoorpAcademy/docker-pyspark/issues/13
Edit file hadoop-2.7.7/etc/hadoop/core-site.xml to contain the following lines:
<configuration><property><name>fs.s3.awsAccessKeyId</name><value>*****</value></property><property><name>fs.s3.awsSecretAccessKey</name><value>*****</value></property></configuration>
b. Copy the following jar files to the hadoop-2.7.7/share/hadoop/common/lib directory
sudo cp hadoop-2.7.7/share/hadoop/tools/lib/aws-java-sdk-1.7.4.jar hadoop-2.7.7/share/hadoop/common/lib/sudo cp hadoop-2.7.7/share/hadoop/tools/lib/hadoop-aws-2.7.5.jar Hadoop-2.7.7/share/hadoop/common/lib/
git clone path/to/your/repo.git
Make sure to add a PYTHONPATH to this folder by adding the following line to your .bashrc file:
export PYTHONPATH=$PYTHONPATH:/home/ubuntu/repo
PYTHONPATH is an environment variable which you can set to add additional directories where python will look for modules and packages. For most installations, you should not set these variables since they are not needed for Python to run. Python knows where to find its standard library.
The only reason to set PYTHONPATH is to maintain directories of custom Python libraries that you do not want to install in the global default location (i.e., the site-packages directory).
Source: https://www.tutorialspoint.com/What-is-PYTHONPATH-environment-variable-in-Python
Happy Sparking!
check out Snipe.gg’s blog:
medium.com
In their case, they found
[...] upon checking the EMR Pricing, we realized that EMR adds a pricing overhead of a whopping 25% to the price of the EC2 instances it uses
This blog also does a great job of highlighting tools and considerations for productionizing the effort I just outlined.