Data Wrangling in the Cloud — Part III — Configuring EMR & Spark Cluster
By: Kelly, Scott
Tags: Spark, Pyspark, Ssh, Aws Emr, S3

tl;dr: For this step, we are going to be provisioning and configuring a 5 node Spark cluster using AWS’ Elastic Map Reduce (EMR) resource.

We need to do a decent amount of preprocessing on the 2.7 million articles. And this is the perfect scenario for a Spark cluster. On your AWS console, navigate to EMR and click “Create cluster”

At the top, click “Advanced Configuration” and you should see something similar to the image below. Be sure to have Spark checked as it is usually not marked by default.

As a part of the configuration step we need to select specific compute instances to put in our cluster. EMR utilizes EC2 instances as the master and compute nodes in the cluster, and with such, there’s a large choice of which type to utilize. Since our data isn’t SUPER huge, we don’t need to go overboard. I will be utlizing the m4.xlarge EC2 instances. They are a 4-core 16 GB compute instance and are especially cheap. As seen below, they’re only $0.20/hr for the compute and $0.06/hr for the EMR configuration. The processing we will be doing is only going to take a few minutes, so this is literally going to cost pennies.

I’ve set my master node with 1 instance of the m4.xlarge compute, while the worker nodes are also set with the same type, but 4 instances of it. Therefore we will have 1 master node and 4 worker nodes in our cluster.

On the next page, be sure to use/create an EC2 key pair so we can SSH into our cluster after it has been provisioned. We will also need this to create an SSH tunnel to utilize jupyter notebooks on our local machine but execute on the cluster. If you create a new one, be sure to download the .pem file and save it locally. For windows users, SSH isn’t a native command. It is possible to install it using chocolatey. But mostly everyone uses puTTy to SSH into remote machines on Windows. HERE is a resource that walks you through converting the .pem file to a .ppk file for PuTTy

That should be the last needed configuration for the cluster. Click “create” and wait for it to start up. It’ll take about 10–15mins for all 5 nodes to be setup and configured. While it is configuring however, Under the summary page for the cluster, you should see a link for the security group in which the cluster has been assigned to. You need to click this link to the security groups page in order to allow an inbound rule. We need to allow SSH rules into our cluster.

After clicking security groups — if you left everything as default — there should be one that says something along the lines of ElasticMapReduce-master. Click that one, and at the bottom you should see a tab that says “Inbound Rules”.

At the bottom right, click “Edit Inbound rules”. In the page that pops up, scroll down and select “Add rule”. A new entry row will pop up. In the drop down menu, select SSH, then in the 4th column where it says “custom”, change it to “my ip”. This opens up port 22 on the master node for SSH tunneling only for your computer from your IP address.

Back on the EMR page in the AWS console, once the cluster has been fully provisioned, you’ll see it say “waiting” at the top. We can now SSH into our master node. For this step, you’ll need the ip address to you cluster’s master (it can be found on the cluster summary page),

and the .pem file for your EC2 Key Pair. You should have downloaded this when you created it while provisioning the cluster. On your local computer, open a terminal and type

ssh -i [path/to/your/.pem/file] hadoop@your_cluster_ip -L 8888:localhost:8888

You can see my usage below. My .pem file is located in my home folder in a hidden directory called .ssh. The hostname for my cluster was hadoop@54.186.77.98. The -L flag says, any request to port 8888 on your local machine is forwarded to port 8888 on the cluster. This is so we can start a jupyter server on the cluster and connect to it from our local machine. I also added port 8889 in my command in the event jupyter decided to run on 8889 on the cluster for whatever reason. You shouldn’t need to add this however.

Once your enter your SSH command successfully, you should see the following.

Next, we need to install a couple of packages on the cluster. We are going to install spark-nlp and jupyter by typing

sudo python3 -m pip install spark-nlp==2.6.3

sudo python3 -m pip install jupyter

Once our modules have been installed we need to configure pyspark to utilize jupyter as the execution driver instead of the pyspark shell. We can do this by simply typing the following:

export PYSPARK_DRIVER_PYTHON=jupyter

export PYSPARK_DRIVER_OPTS='notebook --no-browser --port=8888'

source .bashrc

Note: I utilized port 8889 because I did have another jupyter server running locally on my machine which was already using port 8888.

We are now ready to start our PySpark session and jupyter server at the same. One of the spark packages we are going to be using is spark-nlp, so we need to configure our spark session to utilize this package. It doesn’t exist on the cluster by default, spark will download the package (.jar files) from the maven repository automatically. We do all this by typing the following

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.6.3

If for whatever reason you get the following:

the variable exports aren’t taking, you’re going to have to manually type everything in while invoking pyspark

PYSPARK_DRIVER_PYTHON="jupyter" PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=8888" pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.6.3

The jupyter server is now up and running. Copy one of the links from the server and open a browser on your local machine. Paste the link in the browser and hit enter. You should be greeted by the jupyter main tree.

To ensure proper configuration and that the jars have been downloaded from maven, we can run the following test

If you receive an error from the Tokenizer() output instead of the object reference as seen above, the cluster didn’t properly download the jars. Try restarting the kernel, run the first line only and let it sit for a few minutes. If you look back over at your terminal, you should see it sending commands to the cluster to download the jars from maven. Once you see this, you should be good to go.

And that’s it, we have configured a 5 node Spark cluster to do some distributed processing. Up next, we will read the data from our S3 bucket into the cluster and start preprocessing.