From Pandas to PySpark with Koalas | by Maria Karanasou | Towards Data Science

For those who are familiar with pandas DataFrames, switching to PySpark can be quite confusing. The API is not the same, and when switching to a distributed nature, some things are being done quite differently because of the restrictions imposed by that nature.
I recently stumbled upon Koalas from a very interesting Databricks presentation about Apache Spark 3.0, Delta Lake and Koalas, and thought that it would be nice to explore it.
The Koalas project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.
pandas is the de facto standard (single-node) DataFrame implementation in Python, while Spark is the de facto standard for big data processing. With this package, you can:
- Be immediately productive with Spark, with no learning curve, if you are already familiar with pandas.
- Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).
source: https://koalas.readthedocs.io/en/latest/index.html
Koalas supports ≥ Python 3.5 and from what I can see from the docs, PySpark 2.4.x. Dependencies include pandas ≥ 0.23.0, pyarrow ≥ 0.10 for using columnar in-memory format for better vector manipulation performance and matplotlib ≥ 3.0.0 for plotting.
The different ways to install Koalas are listed here:
koalas.readthedocs.io
But let’s start simple with:
pip install koalas and pip install pyspark
Keep in mind the dependencies mentioned above.
Given the following data:
import pandas as pdfrom databricks import koalas as ksfrom pyspark.sql import SparkSessiondata = {'a': [1, 2, 3, 4, 5, 6],        'b': [100, 200, 300, 400, 500, 600],        'c': ["one", "two", "three", "four", "five", "six"]}index = [10, 20, 30, 40, 50, 60]
You can either start from a pandas DataFrame:
pdf = pd.DataFrame(data, index=index)# from a pandas dataframekdf = ks.from_pandas(pdf)
From a Koalas Dataframe:
# start from raw datakdf = ks.DataFrame(data, index=index)
Or from a spark Dataframe (one way):
# creating a spark dataframe from a pandas dataframesdf2 = spark_session.createDataFrame(pdf)# and then converting the spark dataframe to a koalas dataframekdf = sdf.to_koalas('index')
A full simple example with output:
The API between Pandas and Koalas is more or less the same. More examples in the official documentation:
koalas.readthedocs.io
Some notes on the Koalas project:
If you are starting from scratch with no previous knowledge of Pandas, then diving in straight to PySpark would probably be a better way to learn.
Some functions may be missing — the missing functions are documented here
Some behavior may be different (e.g. Null vs NaN, where NaN is used with Koalas and is more coherent with Pandas and Null with Spark)
Remember that since it is using Spark under the hood, some operations are lazy, meaning they are not really evaluated and executed before there is a Spark action, like printing out the top 20 rows.
I do have some concerns regarding the efficiency of some of the operations, e.g. .to_koalas() gives a No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. warning, which seems to be because of the indexing operation and can be quite problematic depending on the amount of data you have. Note that when you specify the index column name in .to_koalas('index') the warning does not exist, which is reasonable since spark/koalas knows which column to use as index and does not need to bring all the data into one partition in order to calculate a global rank/ index. See more details about this here: https://towardsdatascience.com/adding-sequential-ids-to-a-spark-dataframe-fa0df5566ff6
Disclaimer: I haven’t really used it much, since, when I started learning Spark this wasn’t available, but I really think it is good to know about the available tools and it could be helpful for those coming from the Pandas’ environment — I can still remember the confusion I had from switching from pandas DataFrames to spark Dataframes.
I hope this was helpful and that knowing about Koalas will save you some time and trouble. Any thoughts, questions, corrections and suggestions are very welcome :)
If you want to know more about how Spark works, take a look at:
towardsdatascience.com
About adding indexes in Spark: