How to Start Learning PySpark with no Big Data
By: The Hippie, The Boss
Tags: Programming, Big Data, Analytics, Pyspark, Data Science

A beginner guide for Data Analysts and Data Scientists who need to work with Big Data.

Photo by Danielle MacInnes

Introduction

True, PySpark was created to perform computation on large data sets and it is typically connected to some storage system with gigabytes to terabytes of data (like HDFS).

However, practically speaking, if you are a Data Analyst/Data Scientist in a medium-sized company, you will spend most of your time interacting with an already made Big Data Platform.

Hence, the most efficient first approach to the world of Big Data is to start learning the syntax of PySpark, its main data structure, and some high-level concepts (e.g. Spark’s query planning).

Don’t start by watching hours of YouTube videos about Spark and other big data concepts — It’s fascinating but it won’t help you concretely.

The good news is that:

You can start learning about the language even without having access to a big data platform.

Minimal Set-up (for Mac OS)

If you want to start experimenting with PySpark, the easiest way is to install Docker on your computer and run your code in a Jupyter Notebook inside the container.

In this way, you will not need to install all the required dependencies for PySpark.

Install Docker

Go to DockerHub and install either the “Mac with Intel Chip” or “Mac with Apple chip” version.

You can check which version is correct for your computer by clicking on the apple icon (top-left corner of your screen) and on “About This Mac”. There you will find information about the “Processor” used by your computer.

2. In your Terminal, run the following command to verify that the installation was successful:

$ docker --version

Docker version 20.10.8, build 3967b7d

3. Open the Finder, look for Applications and then for the Docker App. Double-click on the app to launch it and then start the Docker Engine.

4. In the terminal, run the following command to download the Docker image and automatically launch the container:

$ docker run -p 8888:8888 jupyter/pyspark-notebook

Get yourself a cup of coffee - the first time you run this it will take a while ☕️.

Once the command stops printing output, go to the end and click on the URL. It should look similar to this (I say similar as the token changes every time you launch the container):

If you see a page like the one below means that you are all set 🥳. You can now open a new Python 3 shell and start learning PySpark!

Reading data from a CSV file and creating a Spark DataFrame

The easiest way to ingest some data and start practising some PySpark syntax is to import a CSV file and create a data frame.

First, you need to upload your file to the Jupyter notebook environment. Click on Upload and add your CSV file. For example, here we are adding albums.csv

Open a Python 3 Shell by clicking on New and inside the Jupyeter Notebook copy-paste the following code:

Well done! Now you have a PySpark DataFrame and you can start experimenting with the language syntax (e.g. selecting columns, adding new ones, summarizing your data, etc.)!

A few notes on reading a CSV:

sep = | is needed in this case to match the format of our file. Other common separators could be , ; and so on. A good way to understand the format of your file is to check the first lines. You can run !head your_file.csv in a Jupyter notebook cell and see the start of your file.

is needed in this case to match the format of our file. Other common separators could be and so on. A good way to understand the format of your file is to check the first lines. You can run in a Jupyter notebook cell and see the start of your file. header = True means that the file has column names.

means that the file has column names. inferSchema = True makes PySpark go over the data twice: the first time to infer the data types of each column and the second time to ingest the data. This means that importing the data will be slower but data type mismatches will be less frequent. Do it as it is safer!