A Beginner’s Guide to Apache Spark and Python
By: Prashanth Xavier
Tags: Spark, Programming, Java, Apache Spark, Python

A Beginner’s Guide to Apache Spark and Python

What is Apache Spark?

Apache Spark is an open source framework that has been making waves since its inception at UC Berkeley’s AMPLab in 2009; at its core it is a big data distributed processing engine that can scale at will.

Put simply, with growing data, it becomes essential to handle large streaming data and be able to process and perform other operations like machine learning, and Apache Spark does just that. Some experts say that it will become the go-to platform for stream computing in the near future.

It’s often misconceived that it’s a replacement to Hadoop, but it’s only a competitor to Hadoop’s map-reduce framework. Rightfully so, because it is much faster, has one of the shortest learning curve for developers, and coupled with the fact that it is used by top companies in the market, it makes it an easy and effective top-up skill to the resume.

As you can see below, distributed processing is one of the key features, but not the only one.

Since Spark has been very popular, you can expect that a number of vendors provide Spark services in many different ways. Sometimes the options are too vast or overwhelming, so the excitement and desire to learn something new is just lost in trying to explore the right options.

Spoiler Alert!

In this article, we will not dwell on the specifics of Spark and how to use it for streaming data. Instead, it will serve as a compilation of a list of possible options to get started with it. Then I leave the choice to you—you can choose your own adventure!

I recommend that you read this article through before trying the options presented. The idea is to understand the options available and then choose the one that best fits you, and then get hands-on with it. If your interest is specifically in learning Spark, check out the references.

1. Go Local

The first option is the almighty local setup. If you are not a fan of online services, go for this one. It’s local and you have all the control over it, but be aware that it is time-consuming.

If time is not on your side or you do not like the hassle of installations, I’ll save you the trouble, jump ahead and check option 2 and 3.

What you’ll need for local setup: Virtual Box; Ubuntu; and time and patience!

Virtual Box is an application that allows you to run a virtual computer on your PC, and that’s where you will install Ubuntu, a Linux-based operating system, and Spark. If you are already on Ubuntu you can skip this step.

Navigate to the above link, and choose either Windows hosts or OS X hosts depending on your PC, and download the setup. Once downloaded, just double click and follow the instructions with the default settings and you are good to go. Virtual box is now installed.

Jump over to the next link below and download Ubuntu. Ubuntu Desktop is preferred. After this step, you should now have an .iso file downloaded.

Once done, you can hop over to the Virtual box application. It’s basically empty at this point. You can start by adding a New Virtual Machine by clicking on the New button. Give the machine a nice name, choose Linux, and hit next.

After this step you’ll go through a series of options that you can set up for the machine. Firstly, memory size. You can leave it at the recommended size, but depending on your specs, you can assign a reasonable amount of RAM.

Secondly, the hard disk. You can leave it at the recommended 8.00 GB and set an option to Create a virtual hard disk now, and choose VDI (VirtualBox Disk Image) in the hard disk file type window and hit Next.

Thirdly, storage. You can either choose Dynamically allocated or Fixed size. Fixed size is recommended to improve input/output speeds. 20Gb should be a good amount. Hit Create.

Once you hit create, it will take a while. When it’s ready, you will be taken back to the home screen of Virtual box application, but this time with the machine created.

By default it’s Powered Off, but you can double click to Power it On. During the first boot up you will be asked to select a startup disk. This is important and it’s where you will point it to the Ubuntu.iso image that you downloaded earlier. Select Ubuntu.iso and click Start. This will install Ubuntu onto your virtual machine. Along the way, you will be presented with many installation options, and you can choose to customize it or leave it at the defaults. Either way it should be fine, and at the end of it you should have a working operating system—a virtual one.

The first thing you wanna do inside your virtual machine is confirm if Python is already installed. To do this, go into the terminal application in Ubuntu, and type in python3 and hit enter. This should result in a similar output as below.

The version of Python may differ, but as long as it is above 3 it’s fine.

Next, we are gonna do a series of installations which are essential for Spark to run on our virtual machine.

Jupyter Notebook

Installing Jupyter Notebook is one of the easiest ways to interact with Python and build great code. To do this, on the same terminal or a new one, type the following command:

pip3 install jupyter

This should install the Jupyter Notebook system. Once done, you can test it by entering this command on the terminal:

jupyter notebook

This should open up a browser (probably the default one) with the Jupyter Notebook interface. This indicates that our notebook setup is working.

Java

Let’s now install Java, which is essential for Spark to run. On another terminal window, type in these commands one after the other:

sudo apt-get update sudo apt-get install default-jre

The first command will update our apt-get mechanism, after which we install Java using the second command.

Scala

Similarly, let’s install Scala:

sudo apt-get install scala

To test if the installation was successful, you can type in the below command, which should print out the version of Scala installed.

scala -version

Install Py4j

Now we will install a Python library that will connect Java and Scala with Python.

pip3 install py4j

Spark and Hadoop

Now, as we move to the end, we just have to install Spark and Hadoop itself. Navigate to the below link and direct download a Spark release. Make sure to do this step on your virtual machine so it’s downloaded directly onto it.

Open a new terminal, and make sure you are in the same location where the file was downloaded. You can cd into the right folder and run the following command (note that the filename could be different for you based on the version of Spark at the time):

sudo tar -zxvf spark-2.1.0-bin-hadoop2.7.tgz

Basically, this will unzip the package and create the folders we need.

Next, we need to tell Python where to find Spark. It may look like pure sorcery, but just follow along! Do this on the terminal and hit Enter after each line. Watch out for the path to SPARK_HOME, it should be the location where you have unzipped the folder.

export SPARK_HOME=’home/ubuntu/spark-2.1.0-bin-hadoop2.7' export PATH=@SPARK_HOME:$PATH export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH export PYSPARK_DRIVER_PYTHON=”jupyter” export PYSPARK_DRIVER_PYTHON_OPTS=”notebook” export PYSPARK_PYTHON=python3

Phew, that was a lot! If you are still with me we are all set. The icing on the cake can only be the sweet taste of running Spark.

Open a terminal and cd into the following path (path where you installed Spark):

cd /spark-2.1.0-bin-hadoop2.7/python

Once you are in the right directory, open the Jupyter Notebook:

jupyter notebook

Now you should get the browser opened up with the Jupyter Notebook system. Create a new Python notebook, and in an empty cell type the command below and hit CTRL+Enter.

import pyspark

Voila! Our very own virtual machine with Spark installed and running in its full glory. Sparkling!

2. Databricks

Databricks is a platform created by the original creators of Apache Spark, a great way to harness the power of Spark with just a browser. Databricks takes away the burden of tiresome installations and brings computing power directly onto your browser. If you are looking for a quick tour of Spark, this is the best way to go.

If the local setup is a home cooked meal, consider Databricks a gourmet dish.

What you’ll need: a web browser and a strong internet connection.

Although Databricks is aimed at companies transitioning into big data and distributed computing, they do offer a community edition that should serve our purpose.

In order to get started, navigate to the below link and sign up for the community edition.

You will need to validate your email address in order to sign in for the first time. Once in, you will have the possibility to work with a Python notebook setup that is pre-installed.

When you are logged in, click on ‘Create a Blank Notebook’ to get started. You are presented with a Jupyter Notebook where you can type Python code into each cell and run them individually.

Databricks is built for Spark so you don’t need to worry about additional setup. You can instantly type spark in the first cell and hit either Ctrl+Enter or the small play button to the right of the cell.

At the first run, you will be asked to launch and run a cluster. Go ahead and do it, after which you should see something like the above image. You can now create additional cells and continue evaluating Spark, be it ingesting a data set or performing machine learning—it’s up to you.

Bonus: Databricks is cool for all the right reasons, more importantly because it offers a huge data bank to experiment with underneath all of that horsepower.

To take a glance at what is offered, you need to get into the ‘file system’ of Databricks by using a magic command. In a new cell, type %fs followed by ls and run the cell. You should see the dbfs paths listed.

Next, hit ls /databricks-datasets/ to view all the datasets available. If you like any of them, you can easily use it in your code. For instance, I happen to like the simple /people/people.json data and would like to use it in my code. I can simply do this:

data = spark.read.json(“/databricks-datasets/samples/people/people/.json”)

Dive into it and try some of the data sets, you will not be disappointed! Perhaps this is the easiest way to Spark.

3. Google Colab

Google Colaboratory is a free Jupyter Notebook environment that, much like Databricks, runs completely on the cloud, but all the buzz isn’t about the free notebook system but the free GPU. Yes! You heard me right, Colab offers free GPU! This is a big deal for all the deep learning evangelists out there. That’s a story for another season—our focus today is on Spark.

What you’ll need: a web browser; a strong internet connection; a Google account.

Navigate to the link above. You should be logged in through your Google account and presented with a screen to create a ‘New Python 3 Notebook.’ Go ahead and create it.

This step will land us into familiar territory: a Python notebook with an empty cell. Google Colab, unlike Databricks, is not ready-made for Spark, so we need to tweak it a little bit to get started.

If you try to run pyspark in the empty cell, you will get an error.

Let’s fix it together! Google Colab behaves both like a virtual machine and a notebook. Let’s put the following command into an empty cell and hit CTRL+Enter to install Java:

!apt install openjdk-8-jdk-headless -qq > /dev/null

Using ‘!’ at the beginning of the command will simply run it in a shell, indicating that it is not Python code.

Now let’s download Spark and Hadoop.

Once done, you can actually see if the file was downloaded by running !ls -l as you would do on a Linux shell.

As you can see, the file downloaded was a zip file, so let’s unzip it:

!tar xf spark-2.3.3-bin-hadoop2.7.tgz

Now we just need to do a couple of things before getting Sparked: set the Java and Spark variables as below.

import os

os.environ[“JAVA_HOME”] = “/usr/lib/jvm/java-8-openjdk-amd64”

os.environ[“SPARK_HOME”] = “/content/spark-2.3.3-bin-hadoop2.7”

That’s all there is to set up!

Now check out this example code:

The above code initializes Spark by creating a new SparkSession, and then creates a new Spark data frame on the fly using Python list comprehension, and finally the data frame is printed out. Try it in a new cell block.