Pyspark — forecasting with Pandas UDF and fb-prophet | by Alexandre Wrg | Towards Data Science

Before starting anything to work with pandas-udf the prerequisite are
spark ≥ 2.4
pyarrow ≤ 0.14.1 (above this version there’s some issue)
then we need to set up an environment variable for pyarrow to 1. (see import code)
sudo pip3 install pyarrow=0.14.1
Then we can proceed to libraries import.
from pyspark.sql import SparkSessionfrom pyspark.sql.types import *from pyspark.sql.functions import pandas_udf, PandasUDFType, sum, max, col, concat, litimport sysimport os# setup to work around with pandas udf# see answers here https://stackoverflow.com/questions/58458415/pandas-scalar-udf-failing-illegalargumentexceptionos.environ["ARROW_PRE_0_15_IPC_FORMAT"] = "1"from fbprophet import Prophetimport pandas as pdimport numpy as np
If everything going well we can call the spark context through “sc” and see :
Now we can talk about the interesting part, the forecast!
In this tutorial we will use the new features of pyspark: the pandas-udf, like the good old pyspark UDF the pandas-udf is a user-defined function with the goal to apply our most favorite libraries like numpy, pandas, sklearn and more on Spark DataFrame without changing anything to the syntax and return a Spark DataFrame.
In particular, we will explore the GROUPED_MAP attributes of the Pandas UDF which allow us to apply scikit-learn, statsmodels and more to our Spark DataFrame. We will apply the popular Prophet piecewise regression of the Facebook AI to each item | store in our dataset using Pandas UDF without any loops and without any concatenation of results since Prophet does not process multiple time series at the same time.
Pandas_udf can be passed to the base function as a decorator that wraps the whole function with two parameters: the expected output schemas and the GROUPED_MAP attributes.
# define an output schemaschema = StructType([        StructField("store", StringType(), True),        StructField("item", StringType(), True),        StructField("ds", DateType(), True),        StructField("yhat", DoubleType(), True)    ])
With the decorator defined, the rest is very simple and intuitive; we define our function to adjust and forecast our data as we do for a single series.
My function is very simple, I accept as input a Spark dataframe which is the result of the concatenation of the training set and the test set (with the target initialized to null) which is my sales history for any item (the format is a Spark Dataframe). The function will be applied as if there was only one product-store pair and as a Pandas DataFrame so no syntax change, so we define the training and forecast dates and rename the dates and sales columns by the conventions requested by the algorithm, then I call my function that creates a national event dataframe (format: Pandas DataFrame). Then I pass everything in the algorithm, make the forecast and I attach my forecast dataframe which contains only the date and the forecast associated with my initial test set in order to retrieve the product and store id. The whole function is wrapped in pandas-udf and .. that’s it !
N.B: holidays in Prophet argument is just an “hard-coded” holiday pandas DataFrame available in the notebook.
We can train our model.
If all goes well, the console should log each time series that is trained with a few metrics.
Then, each series is predicted, we put it all together in a single Spark DataFrame without concatenation.
The magic works! We have predicted around 50 series of time series in a blink without complication, without for-loop and without retrieval of the results.
In this short tutorial, we see how to use pandas udf to train several models to forecast several time series at a time in a distributed manner. The main advantages of pandas UDF compared to the fact that it allows us to skip the loop and the concatenation part is the computing time we gain by doing it through a distributed framework, each time series is formed on a different partition on a different worker and the whole is gathered at the end, this is a real advantage compared to a loop that can take some time to run for millions of elements with real data.
the notebook is available on my Github here: https://github.com/AlexWarembourg/Medium and the data are available on Kaggle (the link is under comments in the notebook and the main thing. Thanks!