Shop Order Analysis in Python. Product analysis with the Apriori... | by Michael Whittle | Towards Data Science

The Apriori algorithm is used in data mining to identify frequent items and association rule learning in a dataset. This article will focus on one practical and common use case, the analysis of shop orders. For example, which products are ordered more frequently than others, and which unrelated products are ordered as a consequence of ordering another product. To explain this another way maybe a shopper will buy toothpaste when buying a toothbrush or maybe a shopper will buy butter when buying bread. They are not directly related but it’s not unreasonable to find an association between the two. The Apriori algorithm aids in identifying these association rules.
This article will provide a non-maths heavy walkthrough of how it works using the Python mlxtend library. For further reading I recommend Chapter 11 in “Machine Learning in Action” by “Peter Harrington”.
www.manning.com
I’m going to be using a Jupyter notebook to demonstrate this. If you don’t have Jupyter notebooks installed on your local system you can also use Google Colab which has a free cloud version.
I’m going to be using the “mlxtend” Apriori library for Python. If you don’t have it installed you can install it via the command line or via Jupiter notebooks directly.
# from the command line$ python3 -m pip install pandas mlxtend ploty matplotlib networkx# or, from Jupyter notebooks!python3 -m pip install pandas mlxtend ploty matplotlib networkx
As usual the first step in a Jupyter notebook is to load the required libraries.
import pandas as pdimport networkx as nximport plotly.express as pximport matplotlib.pyplot as pltplt.style.use('default')from mlxtend.frequent_patterns import apriorifrom mlxtend.frequent_patterns import association_rules
For the purpose of this walkthrough I’m going to be using a dataset I found on Kaggle called “Market Basket Optimization”. The dataset has a nice mix of products in orders but not presented in a clean way. I had to carry out some pre-processing to make it workable.
df_orders = pd.read_csv('Market_Basket_Optimisation.csv')df_orders.shape(7500, 20)
Our dataset has 7500 rows and 20 features.
I’m not really sure why the author of the dataset presented the data like this.
The first step is to create a negated regular expression. Replace all non-alphanumeric characters and line breaks with underscores.
df_orders = df_orders.replace(r'[^A-z0-9\n]+','_', regex=True)
Then we need to create a new feature called “combined” that has a CSV form of the columns for each row.
x = df_orders.to_string(header=False, index=False, index_names=False).split('\n')vals = [','.join(e.split()) for e in x]df_orders['combined'] = [','.join(e.split()) for e in x]
The problem now is that the NaN entries are also included like this:burgers,meatballs,eggs,NaN,NaN,NaN etc.
The way we remove these entries is using a regular expression.
df_orders_tmp = df_orders[['combined']].replace(to_replace=r',NaN.*$', value='', regex=True)
So the “combined” feature for the first row now looks like:burgers,meatballs,eggs
I also wanted to include an order number feature which just copies the index number.
df_orders_tmp['orderId'] = df_orders_tmp.index
We will now split each order into separate rows by the comma.
df_orders_tmp = df_orders_tmp.assign(combined=df_orders_tmp.combined.str.split(',')).explode('combined').reset_index(drop=True)
We now want to replace those temporary underscores above with spaces.
df_orders_tmp = df_orders_tmp.replace(r'_',' ', regex=True)
If you are wondering why this was done, there were some unexpected characters in the dataset causing problems with this “implode” process. The solution was to just replace them all with underscores, carry out the split, and put the spaces back.
We will need the quantity of each item purchased in the order. It seems straight forward as each column is one item purchased so I will reflect 1 against each row.
df_orders_tmp['itemQuantity'] = 1
Finalise our new dataframe with the columns we require with meaningful names.
df = df_orders_tmp[['orderId','combined','itemQuantity']]df.columns = ['orderId','itemDescription','itemQuantity']
And let’s take a look...
df
df_table = df.copy()df_table['all'] = 'all'fig = px.treemap(df_table.head(30), path=['all', "itemDescription"], values='itemQuantity', color=df_table["itemQuantity"].head(30), hover_data=['itemDescription'], color_continuous_scale='Blues')fig.show()
df.value_counts()
Another really nice visualisation is a network plot to visualise the top 15 products from orders.
df_network = df.copy()df_network_first = df_network.groupby("itemDescription").sum().sort_values("itemQuantity", ascending=False).reset_index()df_network_first["itemType"] = "groceries"df_network_first = df_network_first.truncate(before=-1, after=15) # top 15plt.rcParams['figure.figsize']=(20,20)first_choice = nx.from_pandas_edgelist(df_network_first, source='itemType', target="itemDescription", edge_attr=True)pos = nx.spring_layout(first_choice)nx.draw_networkx_nodes(first_choice, pos, node_size=12500, node_color="lavender")nx.draw_networkx_edges(first_choice, pos, width=3, alpha=0.6, edge_color='black')nx.draw_networkx_labels(first_choice, pos, font_size=18, font_family='sans-serif')plt.axis('off')plt.grid()plt.title('Top 15 Products', fontsize=25)plt.show()
In order to continue we’ll need to sum up multiple products within the same order.
df_grouped = df.groupby(['orderId','itemDescription']).sum()['itemQuantity']
We will now want to unstack the grouped dataframe.
df_basket = df_grouped.unstack().reset_index().fillna(0).set_index('orderId')
The values in the matrix now need to be encoded to 1’s and 0’s. The way we’ll do this is set value to 0 if the value is less than or equal to 0 and 1 if the value is greater than or equal to 1.
def encode_units(x):    if x <= 0:        return 0    if x >= 1:        return 1basket_sets = df_basket.applymap(encode_units)basket_sets.head()
We will use the function, “apriori” from the, “mlxtend” library on, “basket_sets” to create a dataframe, “frequent_itemsets” with the support for each item. We will set the arguments for, “apriori” to “min_support = 0.05” and “use_colnames = True”.
Important terminology
Antecedent (if) is the first part of the Apriori algorithm. This item or product will be found in the dataset. We say “if” because “if” it exists then it’s likely the “Consequent” will as well, the “then”.
Consequent (then) is the second part of the Apriori algorithm. The item or product found in combination with the “Antecedent”.
Association rules are calculated from datasets (for example shop orders), which are made up of two or more items.
Support is an indication of how frequently the item appears in the dataset E.g. how popular an product is in a shop
Confidence is an indication of how often the rule has been found to be true. It indicates how reliable the rule is. E.g. how likely is it that someone would buy toothpaste when buying a toothbrush
Lift is a metric to measure the ratio of the confidence of products occurring together if they were statistically independent. E.g. how likely is another product purchased when purchasing a product, while controlling how popular the other product is. A lift score that is close to 1 indicates that the antecedent and the consequent are independent and occurrence of antecedent has no impact on occurrence of consequent. A Lift score that is greater than 1 indicates that the antecedent and consequent are dependent to each other, and the occurrence of antecedent has a positive impact on occurrence of consequent. A lift score that is smaller than 1 indicates that the antecedent and the consequent are substitute each other that means the existence of antecedent has a negative impact to consequent or visa versa.
Conviction measures the implied strength of the rule from statistical independence. Conviction score is a ratio between the probability that one product occurs without another while they were dependent and the actual probability of one products existence without another. E.g. if the (oranges) → (apples) association has a conviction score of 1.5; the rule would be incorrect 1.5 times more often (50% more often) if the association between the two were totally independent.
frequent_itemsets = apriori(basket_sets, min_support=0.05, use_colnames=True).sort_values(by=['support'], ascending=False)frequent_itemsets.head()
This is telling us that “mineral water”, “eggs”, and “spaghetti” are the most popular items in this shop. The “min_support” argument needs to be fine tuned based on your dataset. 0.05 worked well here but if you try other datasets you may need to decrease it down as far as 0.01 in some cases.
Use the function, “association_rules” from, “mlxtend” to create the dataframe from, “frequent_itemsets” using the arguments metric, “lift”.
rules = association_rules(frequent_itemsets, metric="lift")rules.head()
Just to reiterate, the “antecedent” is the product being purchased and the “consequent” is the product that is often purchased as well with their associated likelihood.
Ideally we are looking for entries where the lift and confidence are high. Something like a “lift” of ≥ 6 and a “confidence” of ≥ 0.8. Just to recap, “lift” is the ratio between the “actual confidence” and the “predicted confidence”. A lift ratio larger than 1.0 implies that the relationship between the two products is more significant than would be expected if the two sets were independent. The larger the lift ratio, the more significant the association. “confidence” is how often the rule has been found to be true. In order words, how reliable this rule is as a percentage.
What we are looking for are rules where the “lift” and “confidence” is highest. This is easily filtered in Pandas. At a minimum we want a “lift” greater than 1 but as high as possible. For the “confidence” I’ve set the rule to anything equal to or higher than 25%.
rules[ (rules['lift'] > 1) & (rules['confidence'] >= 0.25) ]
This is telling us that in this shop if someone buys “spaghetti” they are 34% likely to buy “mineral water” as well.
The Apriori algorithm can be really useful in data mining. The most common use cases are in the retail market, recommender systems, marketing campaigns, user behaviour analysis, etc.
If you enjoyed this, please follow me on Medium
Interested in collaborating? Let’s connect on LinkedIn