Apache Spark Applications with Amazon EMR and S3 Services using Jupyter Notebook
By: Sercan Karagoz
Tags: AWS, Big Data, Aws Emr, Aws S3, Apache Spark

Apache Spark Applications with Amazon EMR and S3 Services using Jupyter Notebook Sercan Karagoz Follow Jan 16 · 7 min read

Technology is developing everyday, even in every second without stopping and it is also improving our lives in many different ways. In the 1990s, 1 megabyte chip ram capacity was a big revolution for Amiga 500 game computers so that we could play even popular games like Street Fighter or Mortal Kombat 😀 with that capacity.

A lot has changed since those years, from 1 megabyte to petabytes, even if to Exabytes… According to the CSIRO, in the next decade, astronomers expect to be processing 10 petabytes of data every hour from the Square Kilometre Array (SKA) telescope. It is really a huge and exciting development including a very important question ;

How will we deal with the massive data?

Ok ok do not panic, we still have the second engine 😀 ( yes i am actually a pilot). Joking aside, today many companies have really big data centers or servers for accessing services like computing power and storage. In addition to that, these types of services can be used as an on-demand delivery over the Internet with time based or power based pricing which we call “ Cloud Computing ”.

Using a Cloud Computing service can be more affordable and efficient instead of buying and maintaining physical data centers and servers.

There are some well-known Cloud Computing services, such as Amazon Web Services, Microsoft Azure and Google Cloud Platform. Today we will use AWS for our Spark application with Jupyter Notebook.

What is AWS?

AWS (Amazon Web Services) is a highly scalable, comprehensive,fast, reliable and user-friendly complete cloud computing platform provided by Amazon. Also it is the most popular as it was the first representer of the cloud computing world. AWS offers many tools such as VPC, EC2, EMR and S3. You can easily sign up from AWS web page without any price.

Today we will use AWS EMR and S3 services in our example. So let’s get to know these services better.

What is EMR?

Amazon Elastic MapReduce (EMR) is an Amazon Web Services (AWS) platform for big data processing and analysis using famous open source tools like Apache Spark, Apache Hive, Apache HBase, Apache Flink and Presto. EMR provides scalable, flexible and fast computing over petabyte-sized large data.

What is S3?

Amazon Simple Storage Service (Amazon S3) offers an object storage service for the internet which is designed to store data (up to 5 terabytes single file) from any kind of source with providing scalability, data availability, security, and performance. Moreover, Amazon claims that S3 provides 99.99percent durability and 99.99 percent availability of objects.

So far I tried to brief you about Cloud Computing, AWS and it’s components EMR and S3. Now it is time for hands-on exercise for better understanding of usage of AWS with our beloved Jupyter Notebook.

***IMPORTANT NOTE: Remember that AWS is a paid service provider, after you sign up and define your credit card, AWS will provide you a pay-as-you-go approach for pricing. As being a new member you will also have free tiers for some services but anyway there will be very small pricing for using EMR services (less than 1 dollar for my example) as long as you do not spend so much time while service is running and do not forget to terminate services after you are done with your computing. I will show you how to terminate your services after we will complete our example.

So let’s start our example;

After signing up for AWS, we will go for the S3 service from the “Storage” list in the AWS Management Console.

In the S3 console, we will select the “Create bucket” button and we can easily create our S3 bucket just after giving a name(you can leave the rest as default) to it by clicking on “Create bucket” at the bottom of the page.

Now we have already created our S3 bucket. You can select the newly created bucket from the S3 console and upload data files inside it. I will upload 2 data files (u.data and u.item) for our example.

We managed to upload data files that we will use for our Spark application successfully. Now we can return to the AWS Management Console and this time we will choose the EMR tool from the “Analytics” list.

In the EMR console, we will select the “Create cluster” button and will start to create our EMR cluster. We will first give a name to our cluster and then we will choose software configuration. I will choose the “emr-5.30.1” release which is compatible with Jupyter Notebook. We will also choose hardware configuration. Normally we have to designate it according to our data size that we are going to deal but today we will choose something quite enough(m4.xlarge) for our example. I will also choose “Proceed without an EC2 key pair” (you can always create EC2 key pair in the EC2 console)and click on the “Create cluster” button at the end.

We will have our new EMR cluster when we will see “Running” in our cluster page.

We can proceed back to the EMR console and we will see that our cluster has been created and waiting for us. So far we have created our S3 bucket and EMR cluster. Now it is time for creating our notebook.

Select “Notebooks” from the left menu and from the new page we will configure our notebook. First we will name it and choose our existing EMR cluster. We will also choose our S3 bucket where our notebook’s file will be saved. Finally we will click on the “Create notebook” button and we will have our notebook when we will see the “Ready” message on the console.

Congratulations!!! Now we have all required tools for our Spark application. We have created the S3 bucket for getting our data and saving back to it, EMR cluster for using Apache Spark and Notebook for opening Jupyter Notebook.

After selecting “Open in Jupyter” button, we will see something so familiar to us which will make us to feel like we are in sweet home again 😀

Let’s select our notebook and start to write our SparkSql queries without forgetting to choose the Pyspark kernel from the “Kernel” menu.

For our example, I will do something quite easy. We have 2 data files about movies. I will try to find out the most rated movie and its name also with its average rating by joining 2 files at the end. Like I mentioned above I will use SparkSQL api for my query.

Of course it is Star Wars 😀… So we did our query , also created a new DataFrame and saved it back to our S3 bucket. Let’s check our bucket again from the S3 console.

As you can see in the screenshot above, we managed to save our data into our bucket as a CSV file.

So we are already done with our project except one more important thing; terminating our EMR cluster.

It is very important to terminate our clusters after we finish our jobs with them. Otherwise, AWS will continue to price it. In the EMR console, tick our EMR cluster that we have created and then select the “Terminate” button. It will be terminated in a short time and will not say; I’ll be back!!! 😀

Furthermore, you can also stop your notebook (notebooks are not paid ) and delete your S3 bucket from the S3 console.

Conclusion

In this article, I have tried to brief how to use AWS for Spark applications with Jupyter Notebook. There are also many more useful tools in AWS which you can always improve your knowledge by searching AWS Documentation in the AWS web page.

I hope you will find this article helpful. In the next article, I will write about how to create and run a Spark standalone cluster locally.

I will be happy to hear any comments or questions from you. May the data be with you!