Market-basket analysis and prediction | by Gokul S Kumar | Towards Data Science

We all have been shopping more from online e-commerce sites recently, probably due to the lock-down imposed in most parts of the world. You must have noticed an up-sell feature named ‘frequently bought together’ in most of these sites, for instance Amazon where it predicts what all items would go-along with the item that you have just added to the cart. Customers are given the choice of adding all the items shown under this feature to the cart or select the required ones. Amazon does it thorough what they call ‘item-to-item collaborative filtering’ where it runs recommendation algorithms based on the item-search history of the customer to improve the shopping experience. In the case of offline retailers, it works pretty much the same. Let us consider the basic example of Bread and Jam. If the retailer finds that there is an increase in the sales of bread, he can further up-sell by giving a discount in the price of jam, so that more customers are bound to buy them together.
This entire process of analyzing the shopping trends of customers is called ‘Market Basket Analysis’. It is an analyzing technique based on the idea that if we buy an item then we are bound to buy or not-buy a group (or single) items. For example, if a customer is buying bread then the chances of him/her buying jam is more. This is represented by the following equation:
This equation is called the Association Mining Rule. This can be thought of as an IF-THEN relationship. If item A is bought by a customer then the chances of item B being bought by the same user in the same transaction is found out. Here A is called the Antecedent and B the consequent. Antecedents are primary item that are found in the basket and consequents are the items that are found with an antecedent/group of antecedents. The metrics for measuring association are:
Support: It tells us about the combination of items bought together frequently. It gives the part of transactions that contain both A and B.
We can filter out the less frequently occurring items-sets using support.
Confidence: It tells us how frequently the items A and B are bought together, for the no. of times A is bought.
Lift: It indicates the strength of a rule over the randomness of A and B being bought together. It basically measures the strength of any association rule(we will talk about association rules below).
More the lift more is the strength of the rule. If the lift is 3 for A -> B then it means if we buy A, the chances of buying B is 3 times.
So the process is to make rules for each item-set to figure out the metrics of its association so as to decide whether to include it in the analysis or not. But consider a large data-set having millions of user and transactions, thereby producing a large amount of item-sets. So to make rules for all of these would be a humongous task. This is where Apriori algorithm enters the scene.
Apriori algorithm uses frequently bought item-sets to generate association rules. It is built on the idea that the subset of a frequently bought items-set is also a frequently bought item-set. Frequently bought item-sets are decided if their support value is above a minimum threshold support value.
To demonstrate the working of this algorithm consider the following transactions:
There are 5 items A,B,C,D and E in total which are bought together in different combinations in each transaction. Lets fix the minimum threshold support value of item-sets formed to be 2.
Iteration-1: Item-sets having 1 item is formed and their support is calculated.
As seen from the support values of each item-set, D has a support of 1 which is less than the threshold value. So we are going to omit that item-set.
Iteration-2: Next we make all the possible item-sets having 2 items in them. All the items in the table F1 are used in this.
Item-sets having support less than 2 are again omitted, {A,B} in this iteration.
Iteration-3: All the possible item-sets having 3 items in them are listed. Then we are going to divide these item-sets into their sub-sets and omit those having a support value less than the threshold i.e., 2. This process is called Pruning.
We will omit {A,B,C} and {A,B,E} as they both contain {A,B} which was omitted in Iteration-2. This pruning section is the key part of Apriori algorithm.
Iteration-4: Using F3 we will create item-sets having 4 items.
As we can see, the support of the only item-set having 4 items is less than 2. So we stop the iterations here and the final item-set is F3.
If I={A,C,E} then the subsets are {A,C}, {A,E}, {C,E}, {A}, {C}, {E}.
If I={B,C,E} the the subsets are {B,C}, {B,E}, {C,E}, {B}, {C}, {E}.
Rules: For filtering out the relevant item-sets we will create certain rules and apply it to the subsets. Assume a minimum confidence value of 60%.
For every subset S of I we make the rule
S →(I-S) (which means if S then I minus S) if
support(I)/support(S) ≥ minimum confidence value i.e., 60%.
Consider {A,C,E}
Rule 1: {A,C} →({A,C,E}- {A,C}) which is {A,C} → {E}
Confidence = support{A,C,E}/support{A,C} = 2/3 = 66.6% > 60%
So rule 1 i.e., {A,C} → {E} is selected.
Rule 2: {A,E} →({A,C,E}- {A,E}) which is {A,E} → {C}
Confidence = support{A,C,E}/support{A,E} = 2/2 = 100% > 60%
So rule 2 i.e., {A,E} → {C} is selected.
Rule 3: {C,E} →({A,C,E}- {C,E}) which is {C,E} → {A}
Confidence = support{A,C,E}/support{C,E} = 2/3 = 66.6% > 60%
So rule 3 i.e., {C,E} → {A} is selected.
Rule 4: {A} →({A,C,E}- {A}) which is {A} → {C,E}
Confidence = support{A,C,E}/support{A} = 2/3 = 66.6% > 60%
So rule 4 i.e., {A} → {C,E} is selected.
Rule 5: {C} →({A,C,E}- {C}) which is {C} → {A,E}
Confidence = support{A,C,E}/support{C} = 2/4 = 50% < 60%
So rule 5 i.e., {C} → {A,E} is rejected.
Rule 6: {E} →({A,C,E}- {E}) which is {E} → {A,C}
Confidence = support{A,C,E}/support{E} = 2/4 = 50% < 60%
So rule 6 i.e., {E} → {A,C} is rejected.
Same steps can be done to {B,C,E}. Lift is not being used as the size of the input data-set is small and there is no need of further filtering. But in the case of larger data set further filtering is done by imposing a minimum lift value for the rules. The values of all three of the association metrics can be tweaked as per requirement.
Lets consider an actual data-set and see how the analysis is done.
The link to the data-set being used is given below
drive.google.com
We are going to sue python and pandas. We are also going to use Mlxtend lib as it contains inbuilt functions for Apriori algorithm and Associaction rules.
Load the necessary libraries:
import pandas as pdfrom mlxtend.frequent_patterns import apriorifrom mlxtend.frequent_patterns import association_rulesimport matplotlib.pyplot as pltfrom matplotlib import styleimport numpy as np
Ingest the data into a pandas data-frame and try to know about the features.
read_df = pd.read_csv(‘transaction_data.csv’)df = read_df.copy()df.info()
Further description of the features are as follows:
UserId — Unique identifier of a user.
TransactionId — Unique identifier of a transaction. If the same TransactionId is present in multiple rows, then all those products are bought together in the same transaction.
TransactionTime — Time at which the transaction is performed
ItemCode — Unique identifier of the product purchased
ItemDescription — Simple description of the product purchased
NumberOfItemsPurchased — Quantity of the product purchased in the transaction
CostPerItem — Price per each unit of the product
Country — Country from which the purchase is made.
EDA and data cleaning is done as follows:
df = df[df.UserId>0] # usedid <=0 : 25%df = df[df.ItemCode>0]df = df[df.NumberOfItemsPurchased>0]df = df[df.CostPerItem>0]df = df[df.ItemDescription.notna()]df = df[df.TransactionTime.str[-4:] != ‘2028’]
The following data-frame is obtained:
EDA:
Lets do some exploratory data analysis now. Lets see the no. of transactions being done in each part of the year.
df.TransactionTime = pd.to_datetime(df.TransactionTime)df[‘month_year’]= pd.to_datetime(df.TransactionTime).dt.to_period(‘M’)df.sort_values(by = [‘month_year’], inplace = True)Ser = df.groupby(‘month_year’).TransactionId.nunique()x = np.arange(0,len(Ser),1)style.use(‘ggplot’)fig = plt.figure(figsize = (10,10))ax1 = fig.add_subplot(111)ax1.plot(x, Ser, color = ‘k’)ax1.fill_between(x, Ser, color = ‘r’, alpha = 0.5)ax1.set_xticks(x)ax1.set_xticklabels(Ser.index)plt.xlabel(‘Time period’)plt.ylabel(‘No. of transactions’)
Basically we are creating a column named month_year to divide the datapoints into the month in which they occurred. Then we are taking the no. of unique transaction occurring in each month and plotting it using matplotlib.
As we can see the as the time advances the online retailer is getting more and more purchases which maxed in the month of Jan’19.
Lets explore the no. of items bought in each transaction.
Ser = df.groupby(‘TransactionId’).ItemDescription.nunique()Ser.describe()
As we can see the min no. of items is 1 and the max is 540. So we need to plot our histogram as shown below:
bins = [0,50,100,150,200,250,300,350,400,450,500,550]fig = plt.figure(figsize = (10,10))plt.hist(Ser, bins, histtype = 'bar', rwidth = 0.5)plt.xlabel('No. of items')plt.ylabel('No. of transactions')plt.show()
Oops! We can see that most of the transactions include items between 0–100 and some between 100–200. The max items transactions shown above probably is an outlier or is a customer who purchases in a large scale. So we need to re-scale our histogram.
bins = [0,10,20,30,40,50,60,70,80,90,100,110,120,130,140,150,160,170,180,190,200]fig = plt.figure(figsize = (10,10))ax1 = fig.add_subplot(111)ax1.hist(Ser, bins, histtype = 'bar', rwidth = 0.5)ax1.set_xticks(bins)plt.xlabel('No. of items')plt.ylabel('No. of transactions')plt.show()
As we can see most transactions included less than 10 items.
Lets find out the most-selling item in the market. This can be interpreted in many way such as the item which brings in the maximum revenue, items which were found in maximum no. of transactions etc. We will be considering the items which bring in maximum revenue.
df[‘total_cost_item’] = df.NumberOfItemsPurchased*df.CostPerItemSer = df.groupby(‘ItemDescription’).total_cost_item.sum()Ser.sort_values(ascending = False, inplace = True)Ser = Ser[:10]fig = plt.figure(figsize = (10,10))ax = fig.add_subplot(111)ax.barh(Ser.index, Ser, height = 0.5)
From the above plot showing the top 10 items, we can see that the item ‘retrospot lamp’ has maximum sales value.
The no. of unique TransactionId’s is found to be 18334 and the no. of unique ItemDescription’s is found to be 3871.
Now that we have our data after all the pre-processing, lets arrange it with the TransactionId’s as the index and the ItemDescriptions as the columns with the total no. of items bought in each transaction of each item as the data-point.
df_set = df.groupby(['TransactionId', 'ItemDescription']).NumberOfItemsPurchased.sum().unstack().reset_index().fillna(0).set_index('TransactionId')
The following data-frame is obtained:
We need to make sure that any positive values are encoded to one and all the negative values(if any) to zero.
def encode(x): if x <= 0: return 0 else: return 1df_set = df_set.applymap(encode)df_set
As the data-frame is ready we can apply Apriori algorithm to get the frequently bought item-sets.
frequent_itemsets = apriori(df_set, min_support = 0.015, use_colnames = True)
Here the minimum threshold support value is set as 1.5%. We get the following item-sets.
We then arrange the itemsets in descending order of their support values which gives us
frequent_itemsets = apriori(df_set, min_support = 0.015, use_colnames = True)top_items = frequent_itemsets.sort_values('support', ascending = False)[:20]for i in range(len(top_items.itemsets)):    top_items.itemsets.iloc[i] = str(list(top_items.itemsets.iloc[i]))fig = plt.figure(figsize = (10,10))ax = fig.add_subplot(111)ax.bar(top_items.itemsets, top_items.support)for label in ax.xaxis.get_ticklabels():    label.set_rotation(90)plt.xlabel('Item')plt.ylabel('Support')
We then apply the association rules to these item-sets formed by Apriori algorithm.
rules = association_rules(frequent_itemsets, metric = 'confidence', min_threshold = 0.2)
The metric used here is confidence and its minimum threshold value is set to be 0.2.
The following data-frame is obtained
All the antecedents with the corresponding consequents are listed with their individual supports, the total support of the item-set and all other metrics.
The summary of the rules data-frames gives us the following insights:
There are a total of 187 rules.The summary of various metrics.
There are a total of 187 rules.
The summary of various metrics.
Lets take the top rules having the highest confidence
top_rules = rules.sort_values(‘confidence’, ascending = False)[:10]
fig = plt.figure(figsize = (10,10))ax = fig.add_subplot(111)ax.scatter(top_rules.support, top_rules.confidence, top_rules.lift)
Plotting the rules in a graph:
import networkx as nxG1 = nx.DiGraph()color_map = []N = 50colors = np.random.rand(N)strs = ['r0', 'r1', 'r2', 'r3', 'r4', 'r5', 'r6', 'r7', 'r8', 'r9']for i in range(10):    G1.add_nodes_from('r'+str(i))    for a in top_rules.iloc[i]['antecedents']:        G1.add_nodes_from([a])        G1.add_edge(a, 'r'+str(i), color = colors[i], weight = 2)    for c in top_rules.iloc[i]['consequents']:        G1.add_nodes_from([c])        G1.add_edge('r'+str(i), c, color = colors[i], weight = 2)for node in G1:    found_a_string = False    for item in strs:        if node == item:            found_a_string = True    if found_a_string:        color_map.append('red')    else:        color_map.append('black')edges = G1.edges()colors = [G1[u][v]['color'] for u,v in edges]weights = [G1[u][v]['weight'] for u,v in edges]pos = nx.spring_layout(G1, k = 16, scale = 1)fig = plt.figure(figsize = (20,20))nx.draw(G1, pos, edges = edges, node_color = color_map, edge_color = colors, width = weights, font_size = 16, with_labels = False)for p in pos:    pos[p][1] += 0.07nx.draw_networkx_labels(G1, pos)
All these plotting and and further analysis can be done more easily in R rather than python as there are more compatible libraries for data-mining and association rules in R.
Further filtering of the obtained item-sets can be done using lift and confidence values (as they measure association rule strength).
Here the mean value of lift and confidence are found to be 9.388 and 0.429 respectively.
rules[(rules.lift >= 9.388) & (rules.confidence >= 0.429)]
The item-sets are filtered such that only those having the lift and confidence values above the mean values are included.
In this way all the required item-sets can be found. Further fine-tuning of the item-sets to find the more probable ones can be done by increasing the threshold values of support and lift. We can store the found item-sets in a csv file for furtherance.
Summary:
We looked at the basic Association Mining Rule and its applications. Then we solved the problem of generating frequently bought item-sets by using Apriori algorithm. The process was applying Association Rules to these generated item-set was looked upon. We also learned how to do this entire process in Python using Pandas and Mlxtend on a comparatively large-scale dataset.