From pandas to PySpark
By: Zolzaya Luvsandorj
Tags: Big Data, Python, Data Science, Pyspark, Pandas

1. Comparisons 🔎

Now, let’s look at the syntax comparisons between the two libraries. Throughout this section, only PySpark outputs will be shown to keep the post less cluttered.

📍 1.1. Basics

Both libraries’ data objects are called DataFrame: pandas DataFrame vs PySpark DataFrame. Let’s import the data and check its shape:

# 🐼 pandas

df = pd.read_csv('penguins.csv')

df.shape # 🎇 PySpark

df = spark.read.csv('penguins.csv', header=True, inferSchema=True)

df.count(), len(df.columns)

When importing data with PySpark, the first row is used as a header because we specified header=True and data types are inferred to a more suitable type because we set inferSchema=True . If you are curious, try importing without these options and inspect the DataFrame and its data type (similar to pandas, you can check data types using df.dtypes for PySpark DataFrames).

Unlike pandas DataFrame, PySpark DataFrame has no attribute like .shape . So to get the data shape, we find the number of rows and columns separately.

Now, let’s check high level information about the data:

# 🐼 pandas

df.info() # 🎇 PySpark

df.printSchema()

While this method doesn’t give identical output to df.info() , it’s one of the closest built-in methods. Time to look at the head of the data:

# 🐼 pandas

df.head() # 🎇 PySpark

df.show(5)

By default, df.show() will show 20 rows if there are more than 20 rows. PySpark DataFrame actually has a method called .head() . Running df.head(5) provides output like this:

Output from .show() method is more succinct so we will be using .show() for the rest of the post when viewing top rows of the dataset. Now let’s look at how to select columns:

# 🐼 pandas

df[['island', 'mass']].head(3) # 🎇 PySpark

df[['island', 'mass']].show(3)

While we can use almost pandas-like syntax here, the following version of snippets are probably more common for selecting columns in PySpark:

df.select('island', 'mass').show(3)

df.select(['island', 'mass']).show(3)

📍 1.2. Filtering

Let’s look at how to filter the data based on a condition:

# 🐼 pandas

df[df['species']=='Gentoo'].head() # 🎇 PySpark

df[df['species']=='Gentoo'].show(5)

The syntax is almost the same between the two libraries. To get the same output, we can also use:

df.filter(df['species']=='Gentoo').show(5) df.filter("species=='Gentoo'").show(5)

Below shows a few common filter comparisons:

# 🐼 pandas

2a df[df['species'].isin(['Chinstrap', 'Gentoo'])].head()

3a df[df['species'].str.match('G.')].head()

4a df[df['flipper'].between(225,229)].head()

5a df[df['mass'].isnull()].head() 1b df.loc[df['species']!='Gentoo'].head()

2b df[~df['species'].isin(['Chinstrap', 'Gentoo'])].head()

3b df[-df['species'].str.match('G.')].head()

4b df[~df['flipper'].between(225,229)].head()

5b df[df['mass'].notnull()].head() 6 df[(df['mass']<3400) & (df['sex']=='Male')].head()

7 df[(df['mass']<3400) | (df['sex']=='Male')].head() # 🎇 PySpark

2a df[df['species'].isin(['Chinstrap', 'Gentoo'])].show(5)

3a df[df['species'].rlike('G.')].show(5)

4a df[df['flipper'].between(225,229)].show(5)

5a df[df['mass'].isNull()].show(5) 1b df[df['species']!='Gentoo'].show(5)

2b df[~df['species'].isin(['Chinstrap', 'Gentoo'])].show(5)

3b df[~df['species'].rlike('G.')].show(5)

4b df[~df['flipper'].between(225,229)].show(5)

5b df[df['mass'].isNotNull()].show(5) 6 df[(df['mass']<3400) & (df['sex']=='Male')].show(5)

7 df[(df['mass']<3400) |(df['sex']=='Male')].show(5)

While both ~ and - work as a negation in pandas, only ~ works as a valid negation in PySpark.

📍 1.3. Sorting

Let’s sort the data and inspect 5 rows with smallest mass:

# 🐼 pandas

df.nsmallest(5, 'mass') # 🎇 PySpark

df[df['mass'].isNotNull()].orderBy('mass').show(5)

Pandas' .nsmallest() and .nlargest() methods sensibly excludes missing values. However, PySpark doesn’t have equivalent methods. To get the same output, we first filter out the rows with missing mass, then we sort the data and inspect the top 5 rows. If there was no missing data, syntax could be shortened to: df.orderBy(‘mass’).show(5) .

Let’s look at another way of sorting using .sort() instead of .orderBy() :

# 🐼 pandas

df.nlargest(5, 'mass') # 🎇 PySpark

df.sort('mass', ascending=False).show(5)

These variations of the syntax also work:

df.sort(df['mass'].desc()).show(5)

df.orderBy('mass', ascending=False).show(5)

df.orderBy(df['mass'].desc()).show(5)

We can sort by multiple columns as follows:

# 🐼 pandas

df.sort_values(['mass', 'flipper'], ascending=False).head() # 🎇 PySpark

df.orderBy(['mass', 'flipper'], ascending=False).show(5)

In PySpark, you can get away without the list like this: df.orderBy(‘mass’, ‘flipper’, ascending=False).show(5) . To sort by multiple columns but in different directions:

# 🐼 pandas

df.sort_values(['mass', 'flipper'], ascending=[True, False]).head() # 🎇 PySpark

df[df['mass'].isNotNull()]\

.sort('mass', 'flipper', ascending=[True, False]).show(5)

Here’s an alternative:

df[df['mass'].isNotNull()]\

.orderBy(df['mass'].asc(), df['flipper'].desc()).show(5)

📍 1.4. Aggregation

Now, let’s look at a few examples to aggregate data. Simple aggregation can be done very similarly as follows:

# 🐼 pandas

df.agg({‘flipper’: ‘mean’}) # 🎇 PySpark

df.agg({'flipper': 'mean'}).show()

When looking at multiple aggregations, we will need to approach differently:

# 🐼 pandas

df.agg({'flipper': ['min', 'max']}) # 🎇 PySpark

from pyspark.sql import functions as F

df.agg(F.min('flipper'), F.max('flipper')).show()

To get distinct values in a column:

# 🐼 pandas

df['species'].unique() # 🎇 PySpark

df.select('species').distinct().show()

To get a number of distinct values in a column:

# 🐼 pandas

df['species'].nunique() # 🎇 PySpark

df.select('species').distinct().count()

📍 1.5. Aggregation by group

By now, you may have noticed PySpark uses camelCase for the methods and functions. This is true for .groupBy() as well. Here’s a simple group by aggregation example:

# 🐼 pandas

df.groupby('species')['mass'].mean() # 🎇 PySpark

df.groupBy('species').agg({'mass': 'mean'}).show()

Here’s an example aggregating multiple selected columns:

# 🐼 pandas

df.groupby(‘species’).agg({‘flipper’: ‘sum’, ‘mass’: ‘mean’}) # 🎇 PySpark

df.groupBy('species').agg({'flipper': 'sum', 'mass': 'mean'}).show()

If we don’t specify a column, it will show stats for all numerical columns:

# 🐼 pandas

df.groupby('species').mean() # 🎇 PySpark

df.groupBy('species').mean().show()

We can also substitute .mean() with .avg() as well. In other words, we can use df.groupBy(‘species’).avg().show() .

That was it for this post! Hope you find these comparisons useful and learned a bit about PySpark syntax. As you may have noticed, there are quite a bit of similarities between the two libraries when it comes to basic tasks. This makes it easier to get started on PySpark for those who have working knowledge in pandas.