A Project-driven Approach to Learning PySpark (Part 1) | by Henok Yemam | Towards Data Science

I have come across several frustrating tutorials on PySpark promising to teach me PySpark in under five minutes. They are click baits and lack the necessary depth to get me started and keep me rolling. So, I decided to write an article in hopes of helping others like myself with a project-driven tutorial as opposed to showing you code snippets and know-hows. I will primarily focus on a list of problems and use PySpark to answer the questions. You may follow along by grabbing the dataset and code here. At the end of this article, I have also included excellent resources I enjoyed learning from. Happy Learning!
Who are the winners of the D1 division in the Germany Football Association (Bundesliga) in the last decade?Which teams have been relegated in the past 10 years?Does Octoberfest affect the performance of Bundesliga?Which season of Bundesliga was the most competitive in the last decade?What's the best month to watch Bundesliga?
Who are the winners of the D1 division in the Germany Football Association (Bundesliga) in the last decade?
Which teams have been relegated in the past 10 years?
Does Octoberfest affect the performance of Bundesliga?
Which season of Bundesliga was the most competitive in the last decade?
What's the best month to watch Bundesliga?
I advise you to start this journey by creating a new conda environment to avoid conflicts and enjoy your learning with fewer obstacles. Below is a code snippet of what I used to install PySpark, you may follow along with the same code if you are using a mac. Otherwise, check this link for easy to follow step-by-step instruction.
$ conda create --name sparkenv$ conda activate sparkenv$ brew update$ brew cask install adoptopenjdk/openjdk/adoptopenjdk8$ pip install pyspark$ pip install findspark
PySpark is an excellent python gateway to the Apache Spark ecosystem. It allows you to parallelize your data processing across distributed nodes or clusters. That may not mean much to you if you are just working on a single laptop and not on the cloud. But that doesn't mean you still can’t take advantage of the API to learn how to integrate it into your data science workflow. I have learned to enjoy PySpark’s smooth integration of SQL with data frames. If you know the basics of SQL and simple pandas dataframe manipulations, then you will certainly appreciate the intuitive PySpark’s SQL function.
Let’s start by importing the necessary modules.
I use SparkSession which is the entry point to programming Spark with the Dataset and DataFrame API (source).
Now the fun begins. Just like Panda’s dataframe, PySpark can open several types of file types. For simplicity, I have chosen to use CSV files.
You may recall panda’s dataframe.head() that shows the top five rows and you may be turned off by the poor aesthetics of PySparks output. You worry not, my friend, there is an easy fix for this by converting the top five rows of your PySpark dataframe to Pandas dataframe. However, I caution you to not be too attached to this command as that defeats the purpose of learning PySpark. This point will come more clear as you get more experience working with PySpark. Notice that the code below and the one after that are self-explanatory with functions like withColumnRenamed, limit, and toPandas. As with all things Python, the syntax is beginner-friendly and intuitive.
Happy now, pandas fans?
Ok, let’s answer the first leading question.
To answer this question, it requires a bit of data manipulation. I have chosen the most straight forward approach so it’s easier to follow. The design is to aggregate the home and away game results separately creating two dataframes: home and away.
Then, inner join them on ‘Team’ and ‘Season’ fields to create a single dataframe containing game level aggregation: table. After that, I use a basic window function to further aggregate game statistics on season level and rank them based on winning percentage and goal differentials. The reason why I added goal differentials is because it’s used as a tie-breaker in soccer.
The result you get at the end is:
Bayern Munich looks like they won the Bundesliga 6 times during 2000–2010! Was the code easy to read and follow? With a bit of SQL knowledge, you could also dive into the data and answer the rest of the leading questions. I have answered them in this notebook, feel free to compare your solution with mine.
Last but not least, as promised earlier, I used the following resources to get myself started with PySpark. I hope you find my project-driven approach to learning PySpark a better way to get yourself started and get rolling. I would advise you to pick a dataset that you like to explore and use PySpark to do your data cleaning and analysis instead of using Pandas. At first, it may be frustrating to keep looking up the syntax. But with a bit of patience, you will soon realize how powerful and intuitive PySpark is. Once you have the experience of working on your own laptop (single node cluster), upgrading to a multi-node cluster databricks environment will be a smoother transition. Happy learning!
Spark By Examples
PySpark SQL Module
Apache Spark For Data Scientist (YouTube)