Pyspark – demand forecasting data science project | by Alexandre Wrg | Towards Data Science

In this article, we will build a step-by-step demand forecasting project with Pyspark. Here, the list of tasks:
Import dataFilter dataFeatures engineering (features creation)Imputing dataFeatures engineering (features transformation)Applying a gradient boosted tree regressorOptimise the model with Kfold and GridSearch MethodOneshot
Import data
Filter data
Features engineering (features creation)
Imputing data
Features engineering (features transformation)
Applying a gradient boosted tree regressor
Optimise the model with Kfold and GridSearch Method
Oneshot
First we will import our data with a predefined schema. I work on a virtual machine on google cloud platform data comes from a bucket on cloud storage. Let’s import it.
from pyspark.sql.types import *schema = StructType([StructField("DATE", DateType()),StructField("STORE", IntegerType()),StructField("NUMBERS_OF_TICKETS", IntegerType()),StructField("QTY", IntegerType()),StructField("CA", DoubleType()),StructField("FORMAT", StringType())])df = spark.read.csv("gs://my_bucket/my_table_in_csv_format", header = 'true', schema=schema)
Then we will apply some filters, we will only work on hypermarkets and make sure that there are no negative quantities or missing date in the data for example.
df = df.filter( (F.col("QTY") > 0) &(F.col("DATE").notNull())& (F.col("DATE").between("2015-01-01", "2019-01-01")) & (~F.col("FORMAT").isin(["PRO", "SUP"])) )
Then, we will define some variables that will be useful for modeling, such as date derivatives.
df = (df .withColumn('yearday', F.dayofyear(F.col("DATE"))) .withColumn('Month', F.Month(F.col('DATE'))) .withColumn('dayofweek', F.dayofweek(F.col('DATE'))) .withColumn('YEAR', F.year(F.col('DATE'))) .withColumn('QUARTER', F.q(F.col('DATE'))) .withColumn('Month', F.Month(F.col('DATE'))) .withColumn('WeekOfYear', F.weekofyear(F.col('DATE'))) .withColumn('Week', F.date_trunc('week',F.col('DATE'))) .withColumn('MonthQuarter', F.when((df[DATE] <= 8), 0)                                 .otherwise(F.when((df['DATE'] <= 16), 1)                                .otherwise(F.when((df['DATE'] <= 24), 2)                                 .otherwise(3)))))
Now we will calculate the reference quantity for each store i.e. the quantity that was sold for the same day 1 year ago. Nevertheless, it may happen that there has been no sale for this day so we will average a 7-day period around this reference date. this function will be more complex and requires numpy. It’s indeed possible to articulate numpy and spark, let’s see how.
First, a user-defined function must be defined to extract the time series from each store in a vectorized and sparse way. We will define one that will create a sparse vector indexed with the days of the year and in values the associated quantities
All right, let’s take a break, he has a few things to explain here.
Input : we ask a list of days index and qty values associated
Output : we return a sparse vector indexed by day that allow us to find qty related to his day index
The whole thing is passed through an UDF and expected to be a VectorUDT to be concise this is a type of vector that can be manipulated by UDF.
Then, to respond to the inputs requested by the function we will create an aggregated dataframe per year, store and apply a collect_list to the days and quantities. As you can see below, we recover all the day-quantity values of the store for its year. These two lists enter our UDF to create the desired vector, now we can work in numpy on this vector!
Now we can define a function that will work on this vector and put it in a UDF again.
and apply it :
df= (df    .join(self_join        , ([self_join.p_id_store == df.STORE, self_join.year_join ==  df.year]),        how = "left"        )        .withColumn("qty_reference", getReference(F.col("yearday"), F.col("qties_vectorized")))        )
Let’s detail this function, first it tries to find the exact reference day, we have attached our dataframe on itself with its previous year so we want the current days to be equal to the index days in the vector and get the value. In case it is not possible to retrieve this value then we will define a window around this key date and average the values of this window.
And finally the result :
But instead of leaving them hard coded like that, we’re going to wrap everything in a Spark Pipeline. To do this we will create a class template that inherits from the Spark Transformer object like the one below and we will repeat this template for each variable. I’m not going to write the whole feature’s pipeline in this article but you can find it in the code on github. We will see at the end of this article how to put together this feature’s pipeline and insert it into the process.
for more details on the features of this class template see my article... : https://towardsdatascience.com/pyspark-wrap-your-feature-engineering-in-a-pipeline-ee63bdb913
Let’s check if there’s some missing values.
df.select([F.count(F.when(F.isnan(c) | F.col(c).isNull(), c)).alias(c) for c in df.columns]).show()
There may not be any more but there could be some after the insertion of our new variables so we will define an Imputer that will come after the variable creation pipeline.
from pyspark.ml.feature import Imputerimputer = Imputer(    inputCols=df.columns,     outputCols=["{}_imputed".format(c) for c in df.columns])#imputer.fit(df).transform(df)
From this table extract below I will show you how we will proceed for the rest of the data transformations before insertion into the forecast algorithm.
We will do the whole process in one go at the very end only.
# needed importfrom pyspark.ml import Pipelinefrom pyspark.ml.feature import PCAfrom pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler
Spark String Indexerencodes a string column of labels to a column of label indices. The indices are in [0, numLabels) the mapping is done by the highest frequency first.
indexers = [ StringIndexer(inputCol=c, outputCol="{0}_indexedd".format(c), handleInvalid = 'error') for c in categorical_col]pip = Pipeline(stages = indexers)fitted_df =pip.fit(df)df = fitted_df.transform(df)
Spark’s OneHotEncoder One-hot encoding maps a categorical feature, represented as a label index, to a binary vector with at most a single one-value indicating the presence of a specific feature value from among the set of all feature values. For string type input data, it is common to encode categorical features using StringIndexer first.
indexers = [ StringIndexer(inputCol=c, outputCol="{0}_indexedd".format(c), handleInvalid = 'error') for c in categorical_col]encoders = [OneHotEncoder(dropLast=True,inputCol=indexer.getOutputCol(),     outputCol="{0}_encodedd".format(indexer.getOutputCol())) for indexer in indexers]pip = Pipeline(stages = indexers + encoders)fitted_df =pip.fit(df)df = fitted_df.transform(df)
Before executing our pipeline, let’s check if there are any missing values that can be annoying in our process. Before running our dimensional reduction algorithm we must pass all our variables in a vector assembler that will return a sparse representation of all our data. Then the PCA algorithm will take this vector to “reduce” it and return another sparse representation in a dataframe column.
We saw how to index, one-hot encode our data, apply an PCA and put everything in a vector ready to be modeled. There are obviously other possibilities to Standardize, normalize...etc. nevertheless the global principle remains the same. Now our feature Inpure have the good shape to be modeled througout a Pyspark Algorithm.
Split dataset
X_train = final_dataset.filter(F.col('DATE').between("2015-01-02", "2018-06-01"))X_test = final_dataset.filter(F.col('DATE') > "2018-06-01")X_train = X_train.withColumn(target, F.log1p(F.col(target)))X_test = X_test.withColumn(target, F.log1p(F.col(target)))
We will train a gradient boosted trees model to do a regression on total qty sold by day by store.
target = 'QTY'gbt = GBTRegressor(featuresCol = 'Features', labelCol=target)fitted = gbt.fit(X_train)yhat = (fitted.transform(X_test)    .withColumn("prediction", F.expm1(F.col("prediction")))    .withColumn(target, F.expm1(F.col(target)))    ).select(F.col("prediction"), F.col("STORE").alias('SID_STORE'), F.col("DATE").alias("ID_DAY")).show()
We will define a python object that we use as a basis to evaluate our model on different metrics.
eval_ = RegressionEvaluator(labelCol= target, predictionCol= "prediction", metricName="rmse")rmse = eval_.evaluate(yhat)print('rmse is %.2f', %rmse)mae = eval_.evaluate(yhat, {eval_.metricName: "mae"})print('mae is %.2f', %mae)r2 = eval_.evaluate(yhat, {eval_.metricName: "r2"})print('R2 is %.2f', %r2)
R2 is 0.84
rmse is 20081.54
mae is 13289.10
Model seems not bad we will try to improve it.
We will try to optimize our GBDT with different parameters and make a kfold to ensure its robustness.
from pyspark.ml.tuning import CrossValidator, ParamGridBuilderparamGrid = (ParamGridBuilder()             .addGrid(gbt.maxDepth, [5, 8, 10, 12])             .addGrid(gbt.maxBins, [32, 64])             .build())cv = CrossValidator(estimator=gbt,                          estimatorParamMaps=paramGrid,                          evaluator=eval_,                          numFolds=3)  cvModel = cv.fit(X_train)yhat = (cvModel.transform(X_test)    .withColumn("prediction", F.expm1(F.col("prediction")))    .withColumn(target, F.expm1(F.col(target)))    )
I only got 0.3 more points but that’s enough! :)
bonus : features importances :)
fi = fitted.featureImportances.toArray()import pandas as pdfeatures = [encoder.getOutputCol() for encoder in encoders] + \ [x +'_imputed' for x in numeric_col] + ['day', 'month', 'weekday', 'weekend', 'monthend', 'monthbegin', 'monthquarter', 'yearquarter']feat_imp = (pd.DataFrame(dict(zip(features, fi)), range(1))  .T.rename(columns={0:'Score'})  .sort_values("Score", ascending =False)  )
N.B : In the features_utils package there are all the classes associated with the features pipeline.
This article concludes my very brief guide on the various bricks of Pyspark for a data science project. I hope they will help even one person in his work. Pyspark is a very powerful tool on large volumes. He does not have the merit of having at his disposal a battery of algorithms like sklearn but he has the main ones and many resources.
Feel free to let me know about anything you would like me to do with Pyspark and thank you !
You can find the code here : https://github.com/AlexWarembourg/Medium