Getting Started with PySpark on AWS EMR | by Brent Lemieux | Towards Data Science

Data Pipelines with PySpark and AWS EMR is a multi-part series. This is part 1 of 2. Check out part 2 if you’re looking for guidance on how to run a data pipeline as a product job.
Getting Started with PySpark on AWS EMR (this article)Production Data Processing with PySpark on AWS EMR (up next)
Getting Started with PySpark on AWS EMR (this article)
Production Data Processing with PySpark on AWS EMR (up next)
If you have been following business and technology trends over the past decade, you’re likely aware that the amount of data organizations are generating has skyrocketed. Businesses are eager to use all of this data to gain insights and improve processes; however, “big data” means big challenges.
Entirely new technologies had to be invented to handle larger and larger datasets. These new technologies include the offerings of cloud computing service providers like Amazon Web Services (AWS) and open-source large-scale data processing engines like Apache Spark. As the amount of data generated continues to soar, aspiring data scientists who can use these “big data” tools will stand out from their peers in the market.
In this guide, I will teach you how to get started processing data using PySpark on an Amazon EMR cluster. This tutorial is for current and aspiring data scientists who are familiar with Python but beginners at using Spark.
Spark is great for processing large datasets for everyday data science tasks like exploratory data analysis and feature engineering. It can also be used to implement many popular machine learning algorithms at scale.
A brief overview of Spark, Amazon S3 and EMRCreating a cluster on Amazon EMRConnecting to our cluster through a Jupyter notebookLoading data from Amazon S3
A brief overview of Spark, Amazon S3 and EMR
Creating a cluster on Amazon EMR
Connecting to our cluster through a Jupyter notebook
Loading data from Amazon S3
From the docs, “Apache Spark is a unified analytics engine for large-scale data processing.” Spark’s engine allows you to parallelize large data processing tasks on a distributed cluster. A Spark cluster contains a master node that acts as the central coordinator and several worker nodes that handle the tasks doled out by the master node.
We’ll be using Python in this guide, but Spark developers can also use Scala or Java. The pyspark.sql module contains syntax that users of Pandas and SQL will find familiar. The pyspark.ml module can be used to implement many popular machine learning models.
Spark uses lazy evaluation, which means it doesn’t do any work until you ask for a result. This way, the engine can decide the most optimal way to execute your DAG (directed acyclical graph — or list of operations you’ve specified). When I define an operation — new_df = df.filter(df.user_action == 'ClickAddToCart') — Spark adds the operation to my DAG but doesn’t execute. Once I ask for a result — new_df.collect() — Spark executes my filter and any other operations I specify.
A quick note before we proceed: using distributed cloud technologies can be frustrating. At first, you’ll likely find Spark error messages to be incomprehensible and difficult to debug. I encourage you to stick with it! Read the errors. Learn what parts are informative and google it. I can’t promise that you’ll eventually stop banging your head on the keyboard, but it will get easier. It wouldn’t be a great way to differentiate yourself from others if there wasn’t a learning curve!
Amazon S3 (Simple Storage Service) is an easy and relatively cheap way to store a large amount of data securely. A typical Spark workflow is to read data from an S3 bucket or another source, perform some transformations, and write the processed data back to another S3 bucket.
Amazon EMR (Elastic Map Reduce) is a big data platform that synchronizes multiple nodes into a scaleable cluster that can process large amounts of data. As mentioned above, we submit our jobs to the master node of our cluster, which figures out the optimal way to run it. The master node then doles out tasks to the worker nodes accordingly.
First things first, create an AWS account and sign in to the console. I recommend taking the time now to create an IAM user and delete your root access keys.
UPDATE: I’ve created an AWS Quick Setup Guide walking you through how to create IAM users and roles, create an S3 bucket, and configure the AWS CLI.
I’ll be using the region US West (Oregon) for this tutorial. You can change your region with the drop-down in the top right:
Warning on AWS expenses: You’ll need to provide a credit card to create your account. To keep costs minimal, don’t forget to terminate your EMR cluster after you are done using it. For this guide, we’ll be using m5.xlarge instances, which at the time of writing cost $0.192 per hour. Also, there is a small monthly charge to host data on Amazon S3 — this cost will go up with the amount of data you host. To avoid continuing costs, delete your bucket after using it.
To install useful packages on all of the nodes of our cluster, we’ll need to create the file emr_bootstrap.sh and add it to a bucket on S3.
#!/bin/bashsudo pip install -U \    matplotlib \    pandas
Navigate to S3 by searching for it using the “Find Services” search box in the console:
Click “Create Bucket”, fill in the “Bucket name” field, and click “Create”:
Click “Upload”, “Add files” and open the file you created emr_bootstrap.sh. Click “Upload” to upload the file.
Navigate to EC2 from the homepage of your console:
Select “Key Pairs”
Click “Create Key Pair” then enter a name and click “Create”.
Your file emr-key.pem should download automatically. Store it in a directory you’ll remember. I put my .pem files in ~/.ssh. Be sure to keep this file out of your GitHub repos, or any other public places, to keep your AWS resources more secure.
Navigate to EMR from your console, click “Create Cluster”, then “Go to advanced options”.
Make the following selections, choosing the latest release from the “Release” dropdown and checking “Spark”, then click “Next”.
Select the “Default in us-west-2a” option “EC2 Subnet” dropdown, change your instance types to m5.xlarge to use the latest generation of general-purpose instances, then click “Next”.
Name your cluster, add emr_bootstrap.sh as a bootstrap action, then click “Next”. The script location of your bootstrap action will be the S3 file-path where you uploaded emr_bootstrap.sh to earlier in the tutorial. Your bootstrap action will install the packages you specified on each node in your cluster.
Select the key pair you created earlier and click “Create cluster”. Your cluster will take a few minutes to start, but once it reaches “Waiting”, you are ready to move on to the next step — connecting to your cluster with a Jupyter notebook.
Navigate to “Notebooks” in the left panel. Click “Create notebook” and follow the step below.
Name your notebook and choose the cluster you just created.
Once your notebook is “Ready”, click “Open”. You’re now ready to start running Spark on the cloud!
In the first cell of your notebook, import the packages you intend to use. For example:
from pyspark.sql import functions as F
You should get the following output:
Note: a SparkSession is automatically defined in the notebook as spark — you will have to define this yourself when creating scripts to submit as Spark jobs.
Next, let’s import some data from S3. We’ll use data Amazon has made available in a public bucket. Let’s look at the Amazon Customer Reviews Dataset. In particular, let’s look at book reviews:
input_bucket = 's3://amazon-reviews-pds'input_path = '/parquet/product_category=Books/*.parquet'df = spark.read.parquet(input_bucket + input_path)df.show()
The /*.parquet syntax in input_path tells Spark to read all .parquet files in the s3://amazon-reviews-pds/parquet/product_category=Books/ bucket directory.
I’ll be coming out with a tutorial on data wrangling with the PySpark DataFrame API shortly, but for now, check out this excellent cheat sheet from DataCamp to get started.
Once you’ve tested your PySpark code in a Jupyter notebook, move it to a script and create a production data processing workflow with Spark and the AWS Command Line Interface. Then, you’re ready to schedule your Spark job on Airflow.
Thank you for reading! Please let me know if you liked the article or if you have any critiques. If this guide was useful to you, be sure to follow me and so you won’t miss any of my future articles.
If you need help with a data project or want to say hi, connect with and message me on LinkedIn. Cheers!