Market Basket Analysis using PySpark
By: Jesko Rehberg
Tags: Pyspark, Data Mining, Market Basket Analysis, Python

Market Basket Analysis using PySpark

FPhoto by Clem Onojeghuo on Unsplash

Motivation:

Do you want to learn how to analyze your customer market baskets regarding frequently bought together items? Look no further, if you are willing to work with PySpark’s FPGrowth.

Solution:

First of all, let us have a look at our dataframe (data is stored in my github repository). In case you want to use Google’s Colab you first need to mount your Google Drive:

# To be able to use your data stored in your Google Drive you first need to mount your Google Drive so you can load and save files to it. from google.colab import drive

drive.mount('/content/gdrive') #You'll need to put in a token which Google will generate for you as soon as you click on the link.

After entering the token your Google Drive is mounted.

Now we load the data into our dataframe and have a look upon:

import pandas as pd

import numpy as np

data = pd.read_excel(‘/content/gdrive/MyDrive/DDDDFolder/DDDD.xlsx’)

data.head()

That’s how our dataframe originally looks like.

So what we have got is Sales per Sales Day, Customer (e.g. Customer code 0 is one specific customer), Sales Item (similar to Customer, Sales Item 0 is one specific Sales Item) and Sales Transaction ID.

We are going to analyse market baskets regarding sales items per sales transaction id. The other columns are of no interest for our data mining.

To install PySpark in Colab we have to use our Superuser Power:

%%capture !sudo apt-get update --fix-missing !apt-get install openjdk-8-jdk-headless -qq > /dev/null !wget -q https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz #!wget -q https://downloads.apache.org/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz !mv spark-3.0.0-bin-hadoop3.2.tgz sparkkk !tar xf sparkkk !pip install -q findspark

Afterwards we can start our Spark session:

import os os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64" os.environ["SPARK_HOME"] = "/content/spark-3.0.0-bin-hadoop3.2" import findspark findspark.init() from pyspark.sql import SparkSession spark = SparkSession.builder.master("local[*]").getOrCreate() spark = SparkSession \ .builder \ .appName('fpgrowth') \ .getOrCreate() spark

You will see this message when everything is up and running.

Different to Pandas, in Spark to create a dataframe we have to use Spark’ s CreateDataFrame:

from pyspark.sql import functions as F

from pyspark.ml.fpm import FPGrowth

import pandas

sparkdata = spark.createDataFrame(data)

For our market basket data mining we have to pivot our Sales Transaction ID as rows, so each row stands for one Sales Transaction ID including the purchased Sales Items. First we make sure that each Sales Transaction ID is unique with regards to its Sales Items (that’s why dropDuplicates). Then we group by Sales Transaction ID and use aggregate function’s collect_list to receive all Sales Items per Sales Transaction ID (row):

basketdata = sparkdata.dropDuplicates(['SalesTransactionID', 'SalesItem']).sort('SalesTransactionID') basketdata = basketdata.groupBy("SalesTransactionID").agg(F.collect_list("SalesItem")).sort('SalesTransactionID')

The result now looks like this and is readily prepared for FPGrowth’s algorithm to work with:

Now we will calculate the Market Basket Measures like Support, Lift, and Confidence. Please note that the parameter settings like MinSupport are very data specific:

#Frequent Pattern Growth – FP Growth is a method of mining frequent itemsets using support, lift, and confidence. fpGrowth = FPGrowth(itemsCol="collect_list(SalesItem)", minSupport=0.006, minConfidence=0.006) model = fpGrowth.fit(basketdata) # Display frequent itemsets. model.freqItemsets.show() items = model.freqItemsets # Display generated association rules. model.associationRules.show() rules = model.associationRules # transform examines the input items against all the association rules and summarize the consequents as prediction model.transform(basketdata).show() transformed = model.transform(basketdata)

That’s how the results look like.

That’s already our FPGrowth result. In case you want to export it as an e.g. Excel file you could do like that:

# Convert the Spark DataFrame back to a Pandas DataFrame using Arrow result_pdf = items.select("*").toPandas()

result_pdf.head()

result_pdf.to_excel('result_pdfItemsFreq.xlsx')

And similar for rules:

rules_pdf = rules.select("*").toPandas()

rules_pdf.head()

rules_pdf.to_excel('rules_pdfAnteConseConfLift.xlsx')

transformed_pdf = transformed.select("*").toPandas()

transformed_pdf.head()

transformed_pdf.to_excel('transformed_pdfSalesTransactionIDCollectListPred.xlsx')

Congratulations, you have successfully conducted a data mining technique using FPGrowth! Maybe you have spotted the column Prediction in the last screenshot? This “Prediction” builds a very nice bridge to recommending systems. Please read my related post if you should be interested.

Hope this was supportive. Any questions, please let me know. You can connect with me on LinkedIn or Twitter. Many thanks for reading!

Originally published on my website DAR-Analytics. The Jupyter Notebook and data file is also available in my Github repository: https://github.com/DAR-DatenanalyseRehberg/DDDD_Data-Driven-Dealings-Development/blob/main/Market_Basket_DDDD.ipynb