Market Basket Analysis using PySpark | by Jesko Rehberg | Towards Data Science

Motivation:
Do you want to learn how to analyze your customer market baskets regarding frequently bought together items? Look no further, if you are willing to work with PySpark’s FPGrowth.
Solution:
First of all, let us have a look at our dataframe (data is stored in my github repository). In case you want to use Google’s Colab you first need to mount your Google Drive:
# To be able to use your data stored in your Google Drive you first need to mount your Google Drive so you can load and save files to it.from google.colab import drivedrive.mount('/content/gdrive')#You'll need to put in a token which Google will generate for you as soon as you click on the link.
Now we load the data into our dataframe and have a look upon:
import pandas as pdimport numpy as npdata = pd.read_excel(‘/content/gdrive/MyDrive/DDDDFolder/DDDD.xlsx’) data.head()
So what we have got is Sales per Sales Day, Customer (e.g. Customer code 0 is one specific customer), Sales Item (similar to Customer, Sales Item 0 is one specific Sales Item) and Sales Transaction ID.
We are going to analyse market baskets regarding sales items per sales transaction id. The other columns are of no interest for our data mining.
To install PySpark in Colab we have to use our Superuser Power:
%%capture!sudo apt-get update --fix-missing!apt-get install openjdk-8-jdk-headless -qq > /dev/null!wget -q https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz#!wget -q https://downloads.apache.org/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz!mv spark-3.0.0-bin-hadoop3.2.tgz sparkkk!tar xf sparkkk!pip install -q findspark
Afterwards we can start our Spark session:
import osos.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"os.environ["SPARK_HOME"] = "/content/spark-3.0.0-bin-hadoop3.2"import findsparkfindspark.init()from pyspark.sql import SparkSessionspark = SparkSession.builder.master("local[*]").getOrCreate()spark = SparkSession \.builder \.appName('fpgrowth') \.getOrCreate()spark
Different to Pandas, in Spark to create a dataframe we have to use Spark’ s CreateDataFrame:
from pyspark.sql import functions as Ffrom pyspark.ml.fpm import FPGrowthimport pandassparkdata = spark.createDataFrame(data)
For our market basket data mining we have to pivot our Sales Transaction ID as rows, so each row stands for one Sales Transaction ID including the purchased Sales Items. First we make sure that each Sales Transaction ID is unique with regards to its Sales Items (that’s why dropDuplicates). Then we group by Sales Transaction ID and use aggregate function’s collect_list to receive all Sales Items per Sales Transaction ID (row):
basketdata = sparkdata.dropDuplicates(['SalesTransactionID', 'SalesItem']).sort('SalesTransactionID')basketdata = basketdata.groupBy("SalesTransactionID").agg(F.collect_list("SalesItem")).sort('SalesTransactionID')
The result now looks like this and is readily prepared for FPGrowth’s algorithm to work with:
Now we will calculate the Market Basket Measures like Support, Lift, and Confidence. Please note that the parameter settings like MinSupport are very data specific:
#Frequent Pattern Growth – FP Growth is a method of mining frequent itemsets using support, lift, and confidence.fpGrowth = FPGrowth(itemsCol="collect_list(SalesItem)", minSupport=0.006, minConfidence=0.006)model = fpGrowth.fit(basketdata)# Display frequent itemsets.model.freqItemsets.show()items = model.freqItemsets# Display generated association rules.model.associationRules.show()rules = model.associationRules# transform examines the input items against all the association rules and summarize the consequents as predictionmodel.transform(basketdata).show()transformed = model.transform(basketdata)
That’s already our FPGrowth result. In case you want to export it as an e.g. Excel file you could do like that:
# Convert the Spark DataFrame back to a Pandas DataFrame using Arrowresult_pdf = items.select("*").toPandas()result_pdf.head()
result_pdf.to_excel('result_pdfItemsFreq.xlsx')
And similar for rules:
rules_pdf = rules.select("*").toPandas()rules_pdf.head()
rules_pdf.to_excel('rules_pdfAnteConseConfLift.xlsx')transformed_pdf = transformed.select("*").toPandas()transformed_pdf.head()
transformed_pdf.to_excel('transformed_pdfSalesTransactionIDCollectListPred.xlsx')
Congratulations, you have successfully conducted a data mining technique using FPGrowth! Maybe you have spotted the column Prediction in the last screenshot? This “Prediction” builds a very nice bridge to recommending systems. Please read my related post if you should be interested.
Hope this was supportive. Any questions, please let me know. You can connect with me on LinkedIn or Twitter. Many thanks for reading!
Originally published on my website DAR-Analytics. The Jupyter Notebook and data file is also available in my Github repository: https://github.com/DAR-DatenanalyseRehberg/DDDD_Data-Driven-Dealings-Development/blob/main/Market_Basket_DDDD.ipynb