Spark Fundamentals for Python Programmers
By: Sruthi Korlakunta
Tags: Big Data, Getting Started, Python, Data Science, Spark

Getting Started

Spark Fundamentals for Python Programmers

Graphic by Author: Spark Bee

I am a python programmer who started to work with PySpark a couple of months ago and realized that it is quite a “big” deal. I really had to work at collecting and understanding some basic terms in order to write my first program in spark. Ergo, I decided to compile this guide for my peers.

This is a condensed article for starting with PySpark if you are making the jump from Python. It is aimed at answering beginners' questions and cleaning out some jargon.

Spark Basics — What is Spark?

Spark is a software application that enables you to handle big data by allowing you to access computational resources from multiple computers.

Keywords: big data, parallel computing, cluster computing, distributed computing, fault-tolerance.

You could use your local computer and python (or excel) to plan your holiday. But if you were to manage and process the Train schedule of the whole country you are going to holiday in, you need something larger. In order to do larger tasks, you also need a larger computational capacity. Spark enables just this. It is an entire framework (Not just a language or server) that allows you to distribute your data and processing over multiple computers.

Graphic by Author: Spark Doodles

With data, a query language, and one computer, you as a data scientist are forced to effectively sample your data such that it is within the limits of your processing power. You then arrive at solutions that are “as representative as possible” of the entire dataset of which they are part. With Spark-based software, you can use the entire meaningful dataset for your analyses. That is basically why the whole thing has been invented — Because data, data everywhere and not a computer to process!

The entire network of computers available for use is called a cluster, each computer in the network is called a node. With Spark, you don't have to bother about deciding which node handles what part of your processing needs. Spark does this implicitly so that you don’t have to bother about threading your process flow.

Before Spark, a popular Open Source library called Hadoop was the core library for big data processing. Since Hadoop’s MapReduce programming model for parallel computing was not as fault-tolerant and had its own limitations in parallel processing, Spark was developed and has been the go-to solution ever since.

Spark is an engine that allows you to program in many languages such as R, Python, Scala, Java, SQL, etc. That is where pySpark comes in. PySpark enables you to use the Spark Framework in python language.

Graphic by Author: If Spark were an Auto

Spark Jargon

Keywords: RDD, Transformations, Actions, Lineage, Lazy Execution, Fault Tolerance, Dataframes, Hadoop File Systems, Spark Session, Spark Context.

RDD: Building blocks for data frames

Graphic by Author : Spark Bee again

RDDs are the core elements of data on which Spark is based. The architecture of an RDD gives spark the ability to process them across multiple machines.

RDD stands for Resilient Distributed Dataset. They do not have a schema (That is, don't imagine them as a row-column table). RDD is a bundle of data elements distributed across various nodes. The name of an RDD is its definition:

Resilient: Resistant to external changes once created. This means that you can perform transformations on an RDD to create new RDDs, but the original RDD remains unchanged. If you have data elements RDD 1 and RDD 2 and perform a transformation using both, you will have a resulting element RDD3, without having changed 1 and 2. Note how this is different from your pandas' execution style. In pandas, you can just modify an existing Dataframe and go on with life. pySpark does not support this. Distributed: RDDs are designed in such a way as to enable distribution onto multiple nodes for parallel data processing. Dataset: They hold data in the form of lists, tuples, etc.

Creating an RDD

You can create an RDD by parallelizing raw data in the Spark Driver Program, or by loading data from external storage such as the .Parquet, Avro etc. For now, don’t bother about the spark context statement in the example. We shall discuss it later.

Creating an RDD by Parallelizing raw data : Example

from pyspark.context import SparkContext sc = sparkContext.getOrCreate( ) parallelData = sc.parallelize([1,2,3,4])

Creating an RDD from Existing File systems: Example

You can load RDDs from existing file formats such as Parquet, Avro, Sequence Files or plain text storage (eg. .txt, .csv)

parallelData = sc.read.parquet(sourcepath+ "filename.parquet")

Lazy Execution, Transformations, and Lineage

As discussed earlier, RDDs are immutable and unchanging after creation. But in order to work with them, you obviously need some control over the data element itself. This is enabled by transformations. Transformations act on existing RDDs and generate new RDDs without changing the original.

Actions are points that return some output to the driver or export data for storage after computation. In spark, transformations are not evaluated as and when they are found, but spark waits until an action is called on it.

Instead of performing all transformations, a lineage graph is made, which records information about dependencies between RDDs without actual data. this Graph is then computed as and when an action is called.

Graphic by Author. Lineage Graph and Lazy Execution: Only the dotted transformations are executed in order to arrive at RDD F when an action is called on RDD F. Irrelevant Transformations are not performed.

This is called Lazy Execution. When an action is found, all and only relevant transformations are performed at once. This saves time and memory which would otherwise be used in computing irrelevant transformations.

Example for lazy execution:

#This is a transformation, it is not performed at this point

parallelData_10 = parallelData.map(lambda x : x*10 ) #This is an action, the above transformation is executed only when this line is executed parallelData_10.collect()

Fault Tolerance

Fault Tolerance is the ability of an RDD to remain functional in case a node in the cluster breaks down. It is the key feature for which RDD was developed and what makes Spark so powerful. The Lineage graph keeps track of the operations performed and helps in data recovery in case an RDD is lost. Either the data is replicated on a different node or it is recovered from the source using the Lineage graph and a different node continues to work on the operation, depending on how the “Manager node” is set up in the cluster.

Spark DataFrames

RDDs do not have a schema. But for many practical purposes, data comes in a semi or fully structured form. Structured Data can be handled in spark using dataframes. You can imagine a spark data frame as being similar to a dataframe in python or a table in RDBMS, with the column-row structure with headers. You can create a spark data frame from an RDD, an existing CSV, .txt file, or a distributed file system such as Avro or Parquet. In general while working with large amounts of data, it makes sense to convert the files into parquet format instead of using text or csv formats.

How is it different from pandas dataframe?

A Spark Dataframe is distributed across the cluster and allows for parallel processing implicitly. Some functions you use with pandas mean different things in the spark.SQL library than in the pandas library (count, describe, etc.). So, I’d watch out for the syntax as a beginner.

Creating a Spark Dataframe

Reading from an existing file (You’ll probably do this a lot. Note the differences from pandas type execution.):

#Feel free to replace csv in this example with parquet, avro, json, myCsv = spark.read.csv(“where/my/file/lives/filename.csv”, inferSchema = True, header = True) #see how your file schema looks, header and datatype

myCSV.printSchema() #see the size myCSV.columns #shows column's names len(myCSV.columns) #shows number columns myCSV.count() #shows rows

Creating a dataframe from scratch (You’ll probably do this only when you learn or experiment):

parallelData = sc.parallelize([(1,2,3,4),(5,6,7,8),(9,10,11,12)]) df = parallelData.toDF(["header1","header2","header3","header4"])

Spark Session and Spark Context

SparkContext connects the Spark driver which runs its executors or nodes (individual computers on the network). It sets up the infrastructure needed to work with RDDs and distributed data processing. This is THE primary entry point to Spark Engine.

Graphic created by the author. Spark Context.

If there are multiple users who want to use the same clusters, and have tables with the same names, but want to keep their work separate from each other, for example, properties of their outputs, transformations, actions, etc. they need their own customized environment. One way of doing this is creating a spark context for each user. They remain independent of each other's spark contexts. since this is a suboptimal and expensive way to spend the resources, Spark 2.0 started to offer a different solution, which is a spark session. Each sparksession remains independent of each other and has its own properties, invisible to other users.

To summarize, spark context is an application. It is what is needed to drive your spark software. It allows your Spark application to access the Spark cluster with the help of the resource manager.

Having multiple SparkSessions is possible thanks to its character. SparkSession is a wrapper for SparkContext. The context is created implicitly by the builder without any extra configuration options.