Speeding Up the Conversion Between PySpark and Pandas DataFrames | Towards Data Science

Converting a PySpark DataFrame to Pandas is quite trivial thanks to toPandas()method however, this is probably one of the most costly operations that must be used sparingly, especially when dealing with fairly large volume of data.
Pandas DataFrames are stored in-memory which means that the operations over them are faster to execute however, their size is limited by the memory of a single machine.
On the other hand, Spark DataFrames are distributed across the nodes of the Spark Cluster which is consisted of at least one machine and thus the size of the DataFrames is limited by the size of the cluster. When there is a need to store more data, you simply need to scale up your cluster by adding more nodes (and/or adding more memory to the nodes).
It is important to understand that when toPandas() method is executed over a Spark DataFrame, all the rows which are distributed across the nodes of the cluster will be collected into the driver program that needs to have sufficient memory to fit the data.
Apache Arrow is a language independent in-memory columnar format that can be used to optimize the conversion between Spark and Pandas DataFrames when using toPandas() or createDataFrame() .
Firstly, we need to ensure that a compatible PyArrow and pandas versions are installed. These are 0.15.1 for the former and 0.24.2 for the latter. More recent versions may also be compatible, but currently Spark does not provide any guarantee so this is pretty much up to the user to test and verify the compatibility. For more details about required versions and compatibility, refer to Spark’s official documentation.
By default, arrow-based columnar data transfers are disabled therefore we have to slightly modify our configurations in order to take advantage of this optimization. To do so, spark.sql.execution.arrow.pyspark.enabled should be enabled.
import numpy as npimport pandas as pd# Enable Arrow-based columnar data spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")# Create a dummy Spark DataFrametest_sdf = spark.range(0, 1000000)# Create a pandas DataFrame from the Spark DataFrame using Arrowpdf = test_sdf.toPandas()# Convert the pandas DataFrame back to Spark DF using Arrowsdf = spark.createDataFrame(pdf)
When an error occurs before the actual computation, PyArrow optimizations will be disabled. If we want to avoid potential fallbacks to non-arrow optimizations, we also need to enable the following configuration:
spark.conf.set(    "spark.sql.execution.arrow.pyspark.fallback.enabled", "true")
Currently, some of the Spark SQL datatypes are not supported by Arrow-based optimization conversions. These types are:
MapType
ArrayType of TimestampType
Nested StructType
As we have already discussed, toPandas() is an expensive operation that should be used carefully in order to minimize the performance impact on our Spark applications. In case where this is required and especially when the dataframe is fairly large, you need to consider PyArrow optimization when converting Spark to Pandas DataFrames (and vice-verca).
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read.