Distributed Processing with PyArrow-Powered New Pandas UDFs in PySpark 3.0 | by Pınar Ersoy | Towards Data Science

Data processing time is so valuable as each minute spent costs back to users in financial terms. This article is mainly for data scientists and data engineers looking to use the newest enhancements of Apache Spark since, in a noticeably short amount of time, Apache Spark has emerged as the next generation big data processing engine, and is highly being practiced throughout the industry faster than ever.
Spark’s consolidated structure supports both compatible and constructible APIs that are formed to empower high performance by optimizing across the various libraries and functions built together in programs enabling users to build applications beyond existing libraries. It gives the opportunity for users to write their own analytical libraries on top as well.
Data is costly to migrate so Spark concentrates on performing computations over the data, regardless of where it locates. In user-interacting APIs, Spark strives to manage these storage systems that seem broadly related in case applications do not require concern about where their data is.
When the data is too big to fit on a single machine with a long time to execute that computation on one machine drives it to place the data on more than one server or computer. This logic requires processing the data in a distributed manner. Spark DataFrame is the ultimate Structured API that serves a table of data with rows and columns. With its column-and-column-type schema, it can span large numbers of data sources.
The purpose of this article is to introduce the benefits of one of the currently released features of Spark 3.0 that is related to Pandas with Apache Arrow usage with PySpark in order to be able to execute a pandas-like UDFs in a parallel manner. In the following headings, PyArrow’s crucial usage with PySpark session configurations, PySpark enabled Pandas UDFs will be explained in a detailed way by providing code snippets for corresponding topics. At the end of the article, references and additional resources are added for further research.
In the previous versions of Spark, there were inefficient steps for converting DataFrame to Pandas in PySpark as collecting all rows to the Spark driver, serializing each row into Python’s pickle format (row by row), and sending them to a Python worker process. At the end of this converting procedure, it unpickles each row into a massive list of tuples. In order to be able to overcome these ineffective operations, Apache Arrow that is integrated with Apache Spark can be used to empower faster columnar data transfer and conversion.
Apache Arrow helps to accelerate converting to pandas objects from traditional columnar memory providing the high-performance in-memory columnar data structures.
Previously, Spark reveals a row-based interface for interpreting and running user-defined functions (UDFs). This introduces high overhead in serialization and deserialization and makes it difficult to work with Python libraries such as NumPy, Pandas which are coded in native Python that enables them to compile faster to machine code.
With the newly proposed UDFs, it advocates introducing new APIs to support vectorized UDFs in Python, in which a block of data is transferred over to Python in some columnar format for execution by serializing block by block instead of row by row.
Pandas package is recognized by machine learning and data science specialists since it has coherent integrations with plenty of Python libraries and packages including scikit-learn, matplotlib, and NumPy.
Also, Pandas UDFs support users both to distribute their data loads and to use the Pandas APIs in Apache Spark.
The user-defined functions can be executed by referring to the official site:
Apache Arrow enables to transfer of data precisely between Java Virtual Machine and executors of Python with zero serialization cost by leveraging the Arrow columnar memory layout to fasten up the processing of string data.
Pandas libraries to work with its instances and APIs.
For better performance, while executing jobs, the following configurations shall be set as follows.
To be able to benefit from PyArrow optimizations, the following configuration can be enabled by setting this config to true which is disabled by default : spark.sql.execution.arrow.pyspark.enabled
The upper enabled optimization may fall back to the non-Arrow optimization implementation situation in case of an error. To cope with this issue that occurs the actual computation within Spark,fallback.enabled shall be set to true : spark.sql.execution.arrow.pyspark.fallback.enabled
Parquet-summary-metadata is not efficient to enable the following configurations for the below reasons:
mergeSchema = false: It is assumed that the schema of all Parquet part-files is identical, for this reason, the footer can be read from any part-files.mergeSchema = true: The footers are required to be read for all files to actualize the merge process.
mergeSchema = false: It is assumed that the schema of all Parquet part-files is identical, for this reason, the footer can be read from any part-files.
mergeSchema = true: The footers are required to be read for all files to actualize the merge process.
spark.sql.parquet.mergeSchema falsespark.hadoop.parquet.enable.summary-metadata false
To sum up, the final recommended list of Arrow optimized configurations are as follows:
"spark.sql.execution.arrow.pyspark.enabled", "true""spark.sql.execution.arrow.pyspark.fallback.enabled", "true""spark.sql.parquet.mergeSchema", "false""spark.hadoop.parquet.enable.summary-metadata", "false"
Proper usage of PyArrow and PandasUDF requires some packages to be upgraded in the PySpark development platform.
The following list of packages is needed to be updated in order to be able to use the latest version of PandasUDF with Spark 3.0 in a proper way.
# Install with Condaconda install -c conda-forge pyarrow# Install PyArrow with Pythonpip install pyarrow==0.15.0# Install Py4j with Pythonpip install py4j==0.10.9# Install pyspark with Pythonpip install pyspark==3.0.0
Also, you may need to assign a new environment variable in order not to face any issues with the PyArrow upgrade of 0.15.1 when running Pandas UDFs.
# Environment Variable Setting for PyArrow Version Upgradeimport osos.environ["ARROW_PRE_0_15_IPC_FORMAT"] = "1"
PyArrow has a greater performance gap when it reads parquet files instead of other file formats. In this blog, you can find a benchmark study regarding different file format reads.
It can be used with different kinds of packages with varying processing times with Python:
Parquet to Arrow : pyarrow.parquet
# Importing PyArrow import pyarrow.parquet as pqpath = "dataset/dimension"data_frame = pq.read_table(path).to_pandas()
Parquet to Arrow with Pandas Dataframe : pyarrow.parquet then convert to pandas.DataFrame
import pandas as pdimport pyarrow as paimport pyarrow.parquet as pqpandas_df = pd.DataFrame(data={'column_1': [1, 2], 'column_2': [3, 4], 'column_3': [5, 6]})table = pa.Table.from_pandas(pandas_df, preserve_index=True)pq.write_table(table, 'pandas_dataframe.parquet')
As long as we are concerned with the performance and processing speed of written scripts, it is beneficial to be aware of how to measure their processing times.
There exist two types of time-passed processing calculation when a Python script is executed.
Processor Time: It measures how long a specific process actively being executed on the CPU. Sleep, waiting for a web request, or time are not included. time.process_time()
Wall-Clock Time: It calculates how much time has passed “on a clock hanging on the wall”, i.e. outside real time.time.perf_counter()
There are additional ways to compute the amount of time spent on a running script.
time.time() function is also quantifes time-passed as a wall-clock time; however it can be calibrated. For this reason, it is needed to go back in time to reset it.
time.monotonic() function is monotonic that simply goes forward; however it has reduced precision performance than time.perf_counter()
Pandas User-Defined Functions can be identified as vectorized UDF that is powered by Apache Arrow permits vectorized operations that serve much higher performance compared to row-at-a-time Python UDFs. They can be accepted as the most impactful improvements in Apache Spark by means of distributed processing of customized functions. They bring countless benefits, including empowering users to use Pandas APIs and improving performance.
Ingesting Spark customized function structures in Python reveals its advanced functionality to SQL users by allowing them to call in the functions without generating the extra scripting effort to connect their functionalities.
Functions can be executed by means of Row, Group, and Window while data formats can be used as Series for column and DataFrame for table structures.
Scalar type of Pandas UDF can be described as the conversion of one or more Pandas Series into one Pandas Series. The final returning data series size is expected to be the same as the input data series.
import pandas as pdfrom pyspark.sql.functions import pandas_udffrom pyspark.sql import Windowdataframe = spark.createDataFrame( [(1, 5), (2, 7), (2, 8), (2, 10), (3, 18), (3, 22), (4, 36)], (“index”, “weight”))# The function definition and the UDF creation@pandas_udf(“int”)def weight_avg_udf(weight: pd.Series) -> float: return weight.mean()dataframe.select(weight_avg_udf(dataframe[‘weight’])).show()
Grouped Agg of Pandas UDF can be defined as the conversion of one or more Pandas Series into one Scalar. The final returned data value type is required to be primitive (boolean, byte, char, short, int, long, float, and double) data type.
# Aggregation Process on Pandas UDFdataframe.groupby("index").agg(weight_avg_udf(dataframe['weight'])).show()w = Window \    .partitionBy('index') \    .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)
# Print the windowed resultsdataframe.withColumn('avg_weight', weight_avg_udf(dataframe['weight']).over(w)).show()
Grouped Map of Pandas UDF can be identified as the conversion of one or more Pandas DataFrame into one Pandas DataFrame. The final returned data size can be arbitrary.
import numpy as np# Pandas DataFrame generationpandas_dataframe = pd.DataFrame(np.random.rand(200, 4))def weight_map_udf(pandas_dataframe):    weight = pandas_dataframe.weight    return pandas_dataframe.assign(weight=weight - weight.mean())dataframe.groupby("index").applyInPandas(weight_map_udf, schema="index int, weight int").show()
According to the specifications of your input and output data, you can switch between these vectorized UDFs by adding more complex functions to them.
The full implementation code and Jupyter Notebook are available on my GitHub.
Questions and comments are highly appreciated!
Python User Defined FunctionsPandas APIApache SparkApache Arrow
Python User Defined Functions
Pandas API
Apache Spark
Apache Arrow
Vectorized UDF: Scalable Analysis with Python and PySparkDemo for Apache Arrow Tokyo Meetup 2018Spark: The Definitive Guide
Vectorized UDF: Scalable Analysis with Python and PySpark
Demo for Apache Arrow Tokyo Meetup 2018
Spark: The Definitive Guide