From pandas to PySpark. Leveraging your pandas data... | by Zolzaya Luvsandorj | Towards Data Science

Being able to skillfully and efficiently manipulate big data is a useful skill to have for data analysts, data scientists and anyone working with data. If you are already comfortable with Python and pandas, and want to learn to wrangle big data, a good way to start is to get familiar with PySpark, a Python API for Apache Spark, a popular open source data processing engine for big data. In this post, we will look at side-by-side comparisons of pandas code snippets for basic data manipulation tasks and their counterparts in PySpark.
This post assumes that the reader is comfortable with manipulating data using pandas in Python.
Let’s start by importing necessary libraries. In PySpark, we will need to create a Spark session. Once the Spark session is created, Spark web user interface (Web UI) can be accessed from: http://localhost:4040/. The application name ‘tutorial’ defined below will be shown as the application name on the right top corner of Web UI. We won’t be using Web UI for the purpose of this post, however, if you are interested in learning more, check out the official documentation.
import pandas as pdfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName('tutorial').getOrCreate()
We will use the penguins dataset for this post. Using the script below, we will save penguins.csv, a modified version of the data, in the working directory.
from seaborn import load_dataset(load_dataset('penguins')    .drop(columns=['culmen_length_mm', 'culmen_depth_mm'])    .rename(columns={'flipper_length_mm': 'flipper',                     'body_mass_g': 'mass'})    .to_csv('penguins.csv', index=False))
Now, let’s look at the syntax comparisons between the two libraries. Throughout this section, only PySpark outputs will be shown to keep the post less cluttered.
Both libraries’ data objects are called DataFrame: pandas DataFrame vs PySpark DataFrame. Let’s import the data and check its shape:
# 🐼 pandas df = pd.read_csv('penguins.csv')df.shape# 🎇 PySparkdf = spark.read.csv('penguins.csv', header=True, inferSchema=True)df.count(), len(df.columns)
When importing data with PySpark, the first row is used as a header because we specified header=True and data types are inferred to a more suitable type because we set inferSchema=True. If you are curious, try importing without these options and inspect the DataFrame and its data type (similar to pandas, you can check data types using df.dtypes for PySpark DataFrames).
Unlike pandas DataFrame, PySpark DataFrame has no attribute like .shape. So to get the data shape, we find the number of rows and columns separately.
Now, let’s check high level information about the data:
# 🐼 pandas df.info()# 🎇 PySparkdf.printSchema()
While this method doesn’t give identical output to df.info() , it’s one of the closest built-in methods. Time to look at the head of the data:
# 🐼 pandas df.head()# 🎇 PySparkdf.show(5)
By default, df.show() will show 20 rows if there are more than 20 rows. PySpark DataFrame actually has a method called .head(). Running df.head(5) provides output like this:
Output from .show() method is more succinct so we will be using .show() for the rest of the post when viewing top rows of the dataset. Now let’s look at how to select columns:
# 🐼 pandas df[['island', 'mass']].head(3)# 🎇 PySparkdf[['island', 'mass']].show(3)
While we can use almost pandas-like syntax here, the following version of snippets are probably more common for selecting columns in PySpark:
df.select('island', 'mass').show(3)df.select(['island', 'mass']).show(3)
Let’s look at how to filter the data based on a condition:
# 🐼 pandas df[df['species']=='Gentoo'].head()# 🎇 PySparkdf[df['species']=='Gentoo'].show(5)
The syntax is almost the same between the two libraries. To get the same output, we can also use:
df.filter(df['species']=='Gentoo').show(5) df.filter("species=='Gentoo'").show(5) 
Below shows a few common filter comparisons:
# 🐼 pandas 2a df[df['species'].isin(['Chinstrap', 'Gentoo'])].head()3a df[df['species'].str.match('G.')].head()4a df[df['flipper'].between(225,229)].head()5a df[df['mass'].isnull()].head()1b df.loc[df['species']!='Gentoo'].head()2b df[~df['species'].isin(['Chinstrap', 'Gentoo'])].head()3b df[-df['species'].str.match('G.')].head()4b df[~df['flipper'].between(225,229)].head()5b df[df['mass'].notnull()].head()6 df[(df['mass']<3400) & (df['sex']=='Male')].head()7 df[(df['mass']<3400) | (df['sex']=='Male')].head()# 🎇 PySpark2a df[df['species'].isin(['Chinstrap', 'Gentoo'])].show(5)3a df[df['species'].rlike('G.')].show(5)4a df[df['flipper'].between(225,229)].show(5)5a df[df['mass'].isNull()].show(5)1b df[df['species']!='Gentoo'].show(5)2b df[~df['species'].isin(['Chinstrap', 'Gentoo'])].show(5)3b df[~df['species'].rlike('G.')].show(5)4b df[~df['flipper'].between(225,229)].show(5)5b df[df['mass'].isNotNull()].show(5)6 df[(df['mass']<3400) & (df['sex']=='Male')].show(5)7 df[(df['mass']<3400) |(df['sex']=='Male')].show(5)
While both ~ and - work as a negation in pandas, only ~ works as a valid negation in PySpark.
Let’s sort the data and inspect 5 rows with smallest mass:
# 🐼 pandas df.nsmallest(5, 'mass')# 🎇 PySparkdf[df['mass'].isNotNull()].orderBy('mass').show(5)
Pandas' .nsmallest() and .nlargest() methods sensibly excludes missing values. However, PySpark doesn’t have equivalent methods. To get the same output, we first filter out the rows with missing mass, then we sort the data and inspect the top 5 rows. If there was no missing data, syntax could be shortened to: df.orderBy(‘mass’).show(5).
Let’s look at another way of sorting using .sort() instead of .orderBy():
# 🐼 pandas df.nlargest(5, 'mass')# 🎇 PySparkdf.sort('mass', ascending=False).show(5)
These variations of the syntax also work:
df.sort(df['mass'].desc()).show(5)df.orderBy('mass', ascending=False).show(5)df.orderBy(df['mass'].desc()).show(5)
We can sort by multiple columns as follows:
# 🐼 pandas df.sort_values(['mass', 'flipper'], ascending=False).head()# 🎇 PySparkdf.orderBy(['mass', 'flipper'], ascending=False).show(5)
In PySpark, you can get away without the list like this: df.orderBy(‘mass’, ‘flipper’, ascending=False).show(5). To sort by multiple columns but in different directions:
# 🐼 pandas df.sort_values(['mass', 'flipper'], ascending=[True, False]).head()# 🎇 PySparkdf[df['mass'].isNotNull()]\  .sort('mass', 'flipper', ascending=[True, False]).show(5)
Here’s an alternative:
df[df['mass'].isNotNull()]\  .orderBy(df['mass'].asc(), df['flipper'].desc()).show(5)
Now, let’s look at a few examples to aggregate data. Simple aggregation can be done very similarly as follows:
# 🐼 pandas df.agg({‘flipper’: ‘mean’})# 🎇 PySparkdf.agg({'flipper': 'mean'}).show()
When looking at multiple aggregations, we will need to approach differently:
# 🐼 pandas df.agg({'flipper': ['min', 'max']})# 🎇 PySparkfrom pyspark.sql import functions as Fdf.agg(F.min('flipper'), F.max('flipper')).show()
To get distinct values in a column:
# 🐼 pandas df['species'].unique()# 🎇 PySparkdf.select('species').distinct().show()
To get a number of distinct values in a column:
# 🐼 pandas df['species'].nunique()# 🎇 PySparkdf.select('species').distinct().count()
By now, you may have noticed PySpark uses camelCase for the methods and functions. This is true for .groupBy() as well. Here’s a simple group by aggregation example:
# 🐼 pandas df.groupby('species')['mass'].mean()# 🎇 PySparkdf.groupBy('species').agg({'mass': 'mean'}).show()
Here’s an example aggregating multiple selected columns:
# 🐼 pandas df.groupby(‘species’).agg({‘flipper’: ‘sum’, ‘mass’: ‘mean’})# 🎇 PySparkdf.groupBy('species').agg({'flipper': 'sum', 'mass': 'mean'}).show()
If we don’t specify a column, it will show stats for all numerical columns:
# 🐼 pandas df.groupby('species').mean()# 🎇 PySparkdf.groupBy('species').mean().show()
We can also substitute .mean() with .avg() as well. In other words, we can use df.groupBy(‘species’).avg().show().
That was it for this post! Hope you find these comparisons useful and learned a bit about PySpark syntax. As you may have noticed, there are quite a bit of similarities between the two libraries when it comes to basic tasks. This makes it easier to get started on PySpark for those who have working knowledge in pandas.
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my article. If you are interested, here are links to some of my other posts:◼️️ 5 tips for pandas users◼️️ 5 tips for data aggregation in pandas◼️️ Writing 5 common SQL queries in pandas◼️️ Writing advanced SQL queries in pandas◼️️ How to transform variables in a pandas DataFrame
Bye for now 🏃 💨