A quick RFM Analysis on Spark
By: Víctor Macedo Becerril
Tags: Retail, Rfm Analysis, Spark, Pyspark, Data Science

Getting retail data

The Retail Data Analytics dataset, shared on Kaggle, will be explored using Spark functions.

First, as I used Databricks Community Edition, is necessary to load the data on the environment.

Import functions:

The functions that will be use are:

import pyspark.sql.functions as sf from pyspark.sql.types import StructType, StructField, IntegerType, StringType, ArrayType, FloatType, BooleanType, DoubleType, DateType from pyspark.sql.functions import to_date, col, datediff, lit

Basically, pyspark.sql.types imported functions are used to declared schemas; pyspark.sql.types imported functions are used to easily operate with the features; and pyspark.sql.functions is to easily use descriptive functions.

Reading data:

First of all, it is necessary to explore the file system, where data was loaded. To do so, lets list the file store. Using Databricks, it is possible using:

display(dbutils.fs.ls("FileStore/tables"))

The result most be something like

To know how to read the files, it is important to make a small visualization, to do so, it is possible to manipulates the Databricks filesystem (DBFS) from the console using the %fs shorthand or using the module dbutils.fs, you can learn more here.

To exemplify, lets use the %fs shorthand.

%fs head FileStore/tables/Features_data_set.csv

with output:

and repeat this process with all the other files.

Once we observed that if the files are comma separated or if they have headers, we proceed to infer the schema.

# File location and type

file_location = "/FileStore/tables/Features_data_set.csv"

file_type = "csv" # CSV options

infer_schema = "true"

first_row_is_header = "true"

delimiter = ","

# The applied options are for CSV files. For other file types, these will be ignored. features = (spark.read.format(file_type)

.option("inferSchema", infer_schema)

.option("header", first_row_is_header)

.option("sep", delimiter)

.load(file_location)) display(features)

As you can see, read a file is so easy. With the display() function, you can see the content of the read file.

It is notable that the schema was not correctly inferred. The method printSchema() allow see the type of each column.

features.printSchema()

Now, we confirm how the shcema was not correctly inferred. The features Date, MarkDown*, CPI and Unemployment are incorrectly inferred so the schema now is declared.

To do so, let’s say to Spark the correct type of each column and read the file again in a new variable (it is important because the variables in Spark are immutable).

As work with dates in Spark is a little complicated, the date was declared as string and then it will be converted as date.

featuresSchema = StructType([

StructField("Store", IntegerType(), True),

StructField("Date", StringType(), True),

StructField("Temperature", DoubleType(), True),

StructField("Fuel_Price", DoubleType(), True),

StructField("MarkDown1", DoubleType(), True),

StructField("MarkDown2", DoubleType(), True),

StructField("MarkDown3", DoubleType(), True),

StructField("MarkDown4", DoubleType(), True),

StructField("MarkDown5", DoubleType(), True),

StructField("CPI", DoubleType(), True),

StructField("Unemployment", DoubleType(), True),

StructField("IsHoliday", BooleanType(), True) ]) features2 = (spark.read

.format(file_type)

.option("header", first_row_is_header)

.option("sep", delimiter)

.schema(featuresSchema)

.load(file_location)) display(features2)

Changing the type of the column Date:

features3 = features2.withColumn("Date",

to_date(col("Date"),

"dd/MM/yyyy")) features3.printSchema()

As it is remarkable, the types and the values of each column now are correctly. This process most also be repeated with the files sales and stores, to get the following dataframes:

Finally, it is for interest know the number of registers of each table with the following commands:

features3.count()

sales3.count()

stores.count()

They gives 8.190, 421.570 and 45 registers respectively.

Joinning dataframes:

Now, lets join our three dataframes. First, take the features3 and sales3 dataframes and full join them on Store, Date and IsHoliday columns; and the result be full joined with stores dataframe on Stores column.

featuresSales = features3.join(sales3,

on=['Store','Date','IsHoliday'],

how='full') featuresSalesStores = featuresSales.join(stores,

on=['Store'],

how='full')

Remember to use the display() function to see if the result is what you expected. On this time, it is.

Given the counts of the records in the features3 (8,190) and sales3 (421,570) tables, it is observed that joining these tables yields (423,325) records, which indicates that there are records in each table that do not have the id corresponding to each other.