Pyspark – Feature engineering (P1) | by Alexandre Wrg | Towards Data Science

In this article, we will see how to calculate new variables via joins, window functions, UDFs and vector manipulations.
As a reminder, here is the table we use :
to create a new column on Spark just pass the function .withColumn and add sql functions
df = (df .withColumn('dayofyear', F.dayofyear(F.col("ID_DAY"))) .withColumn('Month', F.Month(F.col('ID_DAY'))) .withColumn('ratio_ticket_qty', F.col('F_TOTAL_QTY')/F.col('NB_TICKET')) )
For more go here : SQL functions
as with sql it is possible to use the when box for conditional calculations.
df = df.withColumn("day_to_xmas", F.when((F.col("ID_DAY").between("2018-12-01", "2018-12-31")) | (F.col('ID_DAY').between("2019-12-01", "2019-12-31")),     F.lit('xmas_is_coming')).otherwise(F.datediff(F.col("ID_DAY"), F.lit('2018-12-01').cast(DateType()))))
windows function are very useful to calculate values on time axes or to save us joins
grouped_windows = Window.partitionBy(F.col('SID_STORE'), F.col('Month'))rolling_windows = (Window.orderBy(F.col("dayofyear").cast(IntegerType())).rangeBetween(-7, 0))df = (df .withColumn('rolling_average', F.avg("F_TOTAL_QTY").over(rolling_windows)) .withColumn('monthly_qty', F.avg('F_TOTAL_QTY').over(grouped_windows)) )
We can do exactly the same calculation for monthly_qty as with the windows function via a join.
month_qty = df.groupBy('SID_STORE', 'Month').agg(F.avg('F_TOTAL_QTY').alias('monthly_qty_by_join'))df = df.join(month_qty, how = "left", on = ["SID_STORE", "Month"])
We can define functions on pyspark as we would on python but it would not be (directly) compatible with our spark dataframe. To do this, we need to define a UDF (User defined function) that will allow us to apply our function on a Spark Dataframe. The disadvantage is that UDFs can be quite long because they are applied line by line. To apply a UDF it is enough to add it as decorator of our function with a type of data associated with its output.
from pyspark.sql.functions import udf@udf("long")def squared_udf(s):  return s * sdf = df.withColumn('squared_col', squared_udf(F.col('my_columns')))
And that’s it.
Now we will calculate the reference quantity for each store i.e. the quantity that was sold for the same day 1 year ago. Nevertheless, it may happen that there has been no sale for this day so we will average a 7-day period around this reference date. this function will be more complex and requires numpy. Tt is indeed possible to articulate numpy and spark, let’s see how.
First, a user-defined function must be defined to extract the time series from each store in a vectorized and sparse way. We have seen before what a UDF is, we will define one that will create a sparse vector indexed with the days of the year and in values the associated quantities
All right, let’s take a break, he has a few things to explain here.
Input : we ask a list of days index and qty values associated
Output : we return a sparse vector indexed by day that allow us to find qty related to his day index
The whole thing is passed through an UDF and expected to be a VectorUDT to be concise this is a type of vector that can be manipulated by UDF.
Then, to respond to the inputs requested by the function we will create an aggregated dataframe per year, store and apply a collect_list to the days and quantities. As you can see below, we recover all the day-quantity values of the store for its year. These two lists enter our UDF to create the desired vector, now we can work in numpy on this vector!
Now we can define a function that will work on this vector and put it in a UDF again.
and apply it :
df= (df    .join(self_join        , ([self_join.p_id_store == df.SID_STORE, self_join.year_join ==  df.year]),        how = "left"        )        .withColumn("qty_reference", getReference(F.col("yearday"), F.col("qties_vectorized")))        )
Let’s detail this function, first it tries to find the exact reference day, we have attached our dataframe on itself with its previous year so we want the current days to be equal to the index days in the vector and get the value. In case it is not possible to retrieve this value then we will define a window around this key date and average the values of this window.
And finally the result :
Here, another article I wrote on how to create a custom pipeline to integrate all these features in a oneshot :
https://towardsdatascience.com/pyspark-wrap-your-feature-engineering-in-a-pipeline-ee63bdb913
Now you built your amazing feature you need to index your categorical feature or OneHotEncode them and maybe applying some feature reduction and wrap everything in a VectorAssembler to be modeled ! Don’t mind this is the next article :)
You can find the code here : https://github.com/AlexWarembourg/Medium