Run Pandas as Fast as Spark. Why the Pandas API on Spark is a total... | by Adrián González Carpintero | Towards Data Science

That’s it. It’s out. Spark now has a Pandas API.
It seems that, every time you want to work with Dataframes, you have to open a messy drawer where you keep all the tools, and carefully look for the right one.
If you work with structured data, you need SQL. Pandas is always there too. Spark is indispensable for Big Data. There is a tool for every need in the toolbox. But you don’t need the toolbox anymore, because Spark has become the ultimate Swiss Army Knife.
It all started at the 2019 Spark + AI Summit. Koalas, an open source project that enables the use of Pandas on top of Spark, was launched. At the beginning, it only covered a small part of the Pandas functions, but it gradually grew. Two years have passed, and now, in the new Spark 3.2 release, Koalas has been merged into PySpark. And the result is wonderful.
Spark now integrates a Pandas API so you can run Pandas on top of Spark. You just have to change one line of your code:
import pyspark.pandas as ps
Isn’t it great?
Thanks to this, we can obtain a wide range of benefits:
If you use Pandas but you are not familiar with Spark, you can work with Spark right away, with no learning curve.
You can have a single codebase for everything: small data and big data. A single machine and distributed machines.
You can run your Pandas code faster.
This last point is especially remarkable.
On the one hand, you can apply distributed computing to your code in Pandas. But the benefits don’t end there. Thanks to the Spark engine, your code will be faster even in a single machine!
And in case you are wondering, yes, it looks like Pandas-on-Spark is also faster than Dask.
I’m really excited about this breakthrough so, why don’t we get down to business? Let’s do some code with the Pandas API on Spark!
The first thing we need to know is what exactly we are working with. When working with Pandas, we use the class pandas.core.frame.DataFrame. When working with the pandas API in Spark, we use the class pyspark.pandas.frame.DataFrame . Both are similar, but not the same. The main difference is that the former is in a single machine, whereas the latter is distributed.
We can create a Dataframe with Pandas-on-Spark and convert it to Pandas, and vice-versa:
# import Pandas-on-Sparkimport pyspark.pandas as ps# Create a DataFrame with Pandas-on-Sparkps_df = ps.DataFrame(range(10))# Convert a Pandas-on-Spark Dataframe into a Pandas Dataframepd_df = ps_df.to_pandas()# Convert a Pandas Dataframe into a Pandas-on-Spark Dataframeps_df = ps.from_pandas(pd_df)
Note that if you are using multiple machines, when converting a Pandas-on-Spark Dataframe into a Pandas Dataframe, data is transferred from multiple machines to a single one, and vice-versa (see PySpark guide).
We can also convert a Pandas-on-Spark Dataframe into a Spark DataFrame, and vice-versa:
# Create a DataFrame with Pandas-on-Sparkps_df = ps.DataFrame(range(10))# Convert a Pandas-on-Spark Dataframe into a Spark Dataframespark_df = ps_df.to_spark()# Convert a Spark Dataframe into a Pandas-on-Spark Dataframeps_df_new = spark_df.to_pandas_on_spark()
When working with Pandas-on-Spark and Pandas, the data types are basically the same. When converting a Pandas-on-Spark DataFrame to a Spark DataFrame, data types are casted to the appropriate type automatically (see PySpark guide)
The aim of this section is to provide a cheatsheet with the most used functions for managing DataFrames in Spark and their analogues in Pandas-on-Spark. Note that the only difference in syntax between Pandas-on-Spark and Pandas is just the import pyspark.pandas as ps line.
You will see how, even if you are not familiar with Spark, working with it is straightforward thanks to the Pandas API.
# For running Sparkfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName("Spark").getOrCreate()# For running Pandas on top of Sparkimport pyspark.pandas as ps
Let’s use the old dog iris dataset as example.
# SPARKsdf = spark.read.options(inferSchema='True',              header='True').csv('iris.csv')# PANDAS-ON-SPARKpdf = ps.read_csv('iris.csv')
# SPARKsdf.select("sepal_length","sepal_width").show()# PANDAS-ON-SPARKpdf[["sepal_length","sepal_width"]].head()
# SPARKsdf.drop('sepal_length').show()# PANDAS-ON-SPARKpdf.drop('sepal_length').head()
# SPARKsdf.dropDuplicates(["sepal_length","sepal_width"]).show()# PANDAS-ON-SPARKpdf[["sepal_length", "sepal_width"]].drop_duplicates()
# SPARKsdf.filter( (sdf.flower_type == "Iris-setosa") & (sdf.petal_length > 1.5) ).show()# PANDAS-ON-SPARKpdf.loc[ (pdf.flower_type == "Iris-setosa") & (pdf.petal_length > 1.5) ].head()
# SPARKsdf.filter(sdf.flower_type == "Iris-virginica").count()# PANDAS-ON-SPARKpdf.loc[pdf.flower_type == "Iris-virginica"].count()
# SPARKsdf.select("flower_type").distinct().show()# PANDAS-ON-SPARKpdf["flower_type"].unique()
# SPARKsdf.sort("sepal_length", "sepal_width").show()# PANDAS-ON-SPARKpdf.sort_values(["sepal_length", "sepal_width"]).head()
# SPARKsdf.groupBy("flower_type").count().show()# PANDAS-ON-SPARKpdf.groupby("flower_type").count()
# SPARKsdf.replace("Iris-setosa", "setosa").show()# PANDAS-ON-SPARKpdf.replace("Iris-setosa", "setosa").head()
# SPARKsdf.union(sdf)# PANDAS-ON-SPARKpdf.append(pdf)
From now on, you will be able to use Pandas in top of Spark. This leads to an increase in Pandas speed, a decrease in the learning curve when emigrating to Spark, and the merging of single machine computing and distributed computing in the same codebase.
I would like to leave a couple of questions to the reader:
Do you think Spark is going to be the ultimate Swiss Army Knife for managing Dataframes?
Will Pandas kill off other libraries like Dask or Vaex?
Will Pandas users gradually migrate to Spark?
Are we going to seeimport pandas as pdin the future?
Spark User Guide: Pandas API on Spark
Pandas API on Upcoming Apache Spark 3.2