Learning Apache Spark with PySpark & Databricks
By: Todd Birchard
Tags: Big Data, Pyspark, Apache Spark, Python, Data Engineering

Learning Apache Spark with PySpark & Databricks

Something we’ve only begun to touch on so far is the benefit of utilizing Apache Spark is larger-scale data pipelines. Spark is a quintessential part of the Apache data stack: built atop of Hadoop, Spark is intended to handle resource-intensive jobs such as data streaming and graph processing.

Much of Spark’s allure comes from the fact that it is written in Scala & Java. Java and its offshoot languages are notorious for running extremely memory-heavy at run time, which can be used to our advantage. Because our jobs become predictably resource-intensive as everything is stored in memory, this allows us to scale up our Spark Clusters to accommodate accordingly.

A Note About Spark vs. Hadoop

It’s crucial for us to understand where Spark fits in the greater Apache ecosystem. Technically, Spark is built atop of Hadoop: Spark borrows a lot from Hadoop’s distributed file system thus comparing “Spark vs. Hadoop” isn’t an accurate 1-to-1 comparison. Instead, it’s best to think of Spark as being an alternative to Hadoop’s MapReduce.

When compared to Hadoop’s MapReduce, Spark runs faster. Spark stores DataFrames in memory until otherwise stated, thus giving it a speed bonus over MapReduce, which writes to disk. Of course, Spark comes with the bonus of being accessible via Spark’s Python library: PySpark.

If you’re interested in learning how these two play together, I recommend this video which breaks this down quite nicely. If you’re still a bit green to Hadoop’s MapReduce, look no further than this video, which is the best explanation I’ve found to date.

Using Databricks to Get Started

Spinning up a Spark cluster is a topic that deserves a post (or multiple posts) in itself. It’s a non-trivial process that varies per cloud provider and isn’t necessarily the right place to start for those just learning Spark. That’s where Databricks comes in.

Databricks is a private company co-founded from the original creator of Apache Spark. Databricks provides a clean notebook interface (similar to Jupyter) which is preconfigured to hook into a Spark cluster. Thus, we can dodge the initial setup associated with creating a cluster ourselves. Databricks’ free Community Tier is perhaps the best way for anybody to get started learning Spark.

An example Databricks Notebook.

If we wanted, we could very well allow Databricks to own our entire Spark stack, aside from maybe where we keep our final data. Check it out the various tabs that make up the Databricks interface:

Data : This is where we control the input of data. Databricks is heavily integrated with AWS and Azure.

: This is where we control the input of data. Databricks is heavily integrated with AWS and Azure. Clusters : Options for scaling our cluster of servers. Any changes made here are directly reflected in AWS.

: Options for scaling our cluster of servers. Any changes made here are directly reflected in AWS. Jobs : The cool things about Databricks notebooks is that they don’t have to be just notebooks. By linking our notebooks to scheduled jobs, we can use our notebooks as actual scripts to execute regularly. Thus, our notebooks become entirely functional production code as a result.

: The cool things about Databricks notebooks is that they don’t have to be just notebooks. By linking our notebooks to scheduled jobs, we can use our notebooks as actual scripts to execute regularly. Thus, our notebooks become entirely functional production code as a result. Import Library: This one is a bit hard to find… it lives under Databricks > Import library. This is the tool we use to import third-party Python (or Java) libraries for usage in our notebooks. The easiest way to accomplish this is by selecting “PyPI”, and then provide the name of the library (ie: pandas). This will install the library on your cluster and make it available for use.

The easiest way to add some data to work with in Databricks is via their UI. Going back to the data tab, we have the options to hook into an S3 bucket, upload a CSV, or even select from sources such as Amazon Redshift or Apache Kaftka. Unfortunately, Google Cloud is not yet supported by Databricks 😡.

We’ll keep things simple and upload a CSV to kick things off:

Databricks’ interface for importing data.

Upon uploading, we immediately get the option to Create Table in Notebook. It’s best to move forward with this option, as it shows us how to programmatically load data into Spark for the future. Selecting this option will create a new notebook with some boilerplate for us.

Creating Spark DataFrames

There are a few ways to create Spark DataFrames, such as from CSVs, JSON files, or even by stitching together RDDs. Since we’ve already uploaded a CSV, we’ll start with CSVs.

DataFrames from CSVs

If you selected the Create Table in Notebook option when you uploaded your CSV, you should see a notebook with a first cell that looks like this:

# File location and type

file_location = "/FileStore/tables/posts.csv"

file_type = "csv" # CSV options

infer_schema = "true"

first_row_is_header = "true"

delimiter = "," # The applied options are for CSV files. For other file types, these will be ignored.

df = spark.read.format(file_type) \

.option("inferSchema", infer_schema) \

.option("header", first_row_is_header) \

.option("sep", delimiter) \

.load(file_location) display(df)

This cell is showing us the preferred method for creating a DataFrame from a CSV file. Notice how we have a few options chained together to determine how our CSV should be created:

infer_schema will attempt to look at patterns in the data we’ve uploaded and automatically downcast each column to the proper data type. This has proven to be surprisingly accurate; let’s preview this functionality by setting the value to “true”.

will attempt to look at patterns in the data we’ve uploaded and automatically downcast each column to the proper data type. This has proven to be surprisingly accurate; let’s preview this functionality by setting the value to first_row_is_header does exactly what you’d assume. We should also set this to “true”.

does exactly what you’d assume. We should also set this to delimiter allows us to separate columns by a character other than a comma. Leave this as “,”, unless your Comma Separated Values are separated by something other than commas, you madman.

We’ll notice a few things that might look familiar, as well as a few quirks:

Boolean values in PySpark are set by strings (either “true” or “false” , as opposed to True or False ). This is presumably an artifact of Java/Scala, as our Python code is translated into Java jobs.

or , as opposed to or ). This is presumably an artifact of Java/Scala, as our Python code is translated into Java jobs. Next is the presence of df , which you’ll recognize as shorthand for DataFrame . PySpark has its own implementation of DataFrames. These are conceptually similar to what we already know with Pandas, but the syntax and underlying tech are quite different.

, which you’ll recognize as shorthand for . PySpark has its own implementation of DataFrames. These are conceptually similar to what we already know with Pandas, but the syntax and underlying tech are quite different. Finally, display() takes the place of print(). As we work, you’ll find that Databricks has some very cool ways of displaying data previews.

Our first cell is now ready to rock. Let’s see what running this does:

Our data loaded inline.

Our data is now loaded into memory!

DataFrames from JSON

Spark also has a built-in method for reading JSON files into DataFrames. Here’s the JSON I’ll be loading:

[{

"id": 1,

"player_name": "Nick Young",

"team_name": "Lakers"

},

{

"id": 2,

"player_name": "Lou Williams",

"team_name": "Lakers"

},

{

"id": 13,

"player_name": "Lebron James",

"team_name": "Cavaliers"

},

{

"id": 4,

"player_name": "Kyrie Irving",

"team_name": "Cavaliers"

}]

Creating a DataFrame from this JSON file is quite simple:

df = spark.read.json("/FileStore/tables/nbaplayers.json", multiLine=True)

display(df)

Because our JSON object spans across multiple lines, we need to pass the multiLine parameter (I’ve actually found that pretty much all JSON objects will fail unless multiLine is set to True ever since Spark 2.3, and I’m not quite sure why). Take note of the capitalization in “multiLine”- yes it matters, and yes it is very annoying.

Here’s how our data came out:

Dealing With Nested JSON

Quite often we’ll need to deal with JSON structured with nested values, like so:

{

"data": [

[1, "Nick Young", "Lakers"],

[2, "Lou Williams", "Lakers"],

[13, "Lebron James", "Cavaliers"],

[4, "Kyrie Irving", "Cavaliers"],

[5, "Chris Paul", "Lakers"],

[6, "Blake Griffin", "Lakers"],

[20, "Kevin Durant", "Warriors"],

[17, "Stephen Curry", "Warriors"],

[9, "Dwight Howard", "Lakers"],

[10, "Kyle Korver", "Lakers"],

[11, "Kyle Lowry", "Cavaliers"],

[2, "DeMar DeRozan", "Cavaliers"]

]

}

Creating a table of this as-is doesn’t give much of anything useful; we only get a single column:

The first step we can take here is using Spark’s explode() function. explode() splits multiple entries in a column into multiple rows:

from pyspark.sql.functions import explode explodedDF = df.select(explode("data").alias("d"))

display(explodedDF)

The above creates a new DataFrame by “exploding” the data column:

This only gets us halfway there- we still need to deal with the fact that we only have 1 column for three values. Since our column is an array, we can access each item fairly easily by getting the value at each index:

import pyspark.sql.functions as f newPlayersDF = explodedDF.select(explodedDF.d[0], explodedDF.d[1], explodedDF.d[2])

newPlayersDF = explodedDF.selectExpr("d[0] as id", "d[1] as name", "d[2] as team")

display(newPlayersDF)

Saving Our Data to Disk

While most of the work we do will revolve around modifying data in memory, we’ll eventually want to save our data to disk. We can do this by either creating a temporary view, which is only accessible by this notebook, or by creating a persistent table which can be accessed across views.

Creating a temporary view is as simple as:

# Create a temporary view or table temp_table_name = "sampledata" df.createOrReplaceTempView(temp_table_name)

Alternatively, creating a persistent table looks like this:

# Create a permanent table permanent_table_name = "JiraIssue_csv" df.write.format("parquet").saveAsTable(permanent_table_name)

Writing SQL in Databricks

You may have noticed that the auto-generated notebook contains a cell which begins with %sql, and then contains some SQL code. % denotes a magic function: specifically in this case, its a magic function which sets the programming language the cell will be written in. Cells which begin with %sql will be run as SQL code. We created a “Python” notebook thus %python is the default, but %scala, %java, and %r are supported as well.

Writing SQL in a Databricks notebook has some very cool features. For example, check out what happens when we run a SQL query containing aggregate functions as per this example in the SQL quickstart notebook:

Oh snap, we immediately see a visual of the aggregate we just ran!

We could easily spend all day getting into the subtleties of Databricks SQL, but that’s a post for another time.

Spark DataFrame Basics

DataFrames are the bread and butter of how we’ll be working with data in Spark. Before we go further into Spark DataFrames, I’m obligated to mention three essential truths:

Immutable : Spark DataFrames like to be created once upfront, without being modified after the fact. As a result, the way we typically transform DataFrames is by creating a new DataFrame from the original, with transformations applied.

: Spark DataFrames like to be created once upfront, without being modified after the fact. As a result, the way we typically transform DataFrames is by creating a new DataFrame from the original, with transformations applied. Lazy evaluation : DataFrame-based tasks are not executed until explicitly executed. This shouldn’t be a new concept, as those familiar with the PEP style guide know that most data-related tasks in Python are executed this way.

: DataFrame-based tasks are not executed until explicitly executed. This shouldn’t be a new concept, as those familiar with the PEP style guide know that most data-related tasks in Python are executed this way. Distributed: Spark DataFrames are fault-tolerant and highly-available, much like vanilla Hadoop. Thus, we are at little risk of something going horribly wrong and wiping our DataFrame from existence due to external factors — if a node in our Spark cluster goes down, Spark can charge forward regardless.

Great, but what does working with a Spark DataFrame look like? The general structure of modifying a Spark DataFrame typically looks like this:

new_df = original_df.

transformation_1(original_df).

transformation_2(original_df).

transformation_3(original_df)

As we mentioned before, Spark DataFrames are immutable, so we need to create a new DataFrame from our original each time we’d like to make adjustments (AKA: new_df = original_df.etc() ).

From there, we can chain together transformations to ensure we don’t create a new DataFrame per transformation, which would be a ridiculous waste of memory.

Selecting Data in a DataFrame

The last thing we’ll cover is how to select data matching criteria from a DataFrame. The filter() and where() methods all us to select only the rows of a DataFrame which match the conditional statement we pass.

The below will return a DataFrame which only contains rows where the author column has a value of todd:

filter_df = explode_df.filter(explode_df.author == "todd") display(filter_df)

We can also use sort() to order our results:

filter_df = explode_df.filter(explode_df.author == "todd").sort(explode_df.published_at) display(filter_df)

.where() is another way of achieving the same effect:

where_df = explode_df.where((col("firstName") == "todd")).sort(desc("published_at")) display(where_df)

Not the difference in syntax between filter() and select() !

Get Your Hands Dirty

As much as I love throwing function definitions your way, attempting to memorize functions is no way to learn. At this point, I’d recommend playing around in a DataBricks notebook of your own before going much further.

We have a whole lot more ground to cover; we’ve only barely started to scratch the surface of what we can do with PySpark. We still need to create our own custom transformations and much, much more. Stick around for part 2!