Market Basket Analysis with Pandas | by Soner Yıldırım | Towards Data Science

Market basket analysis is a common data science practice implemented by retailers. The goal is to discover the associations among items. It is very important to have an idea of what people tend to buy together.
Having a decent market basket analysis provides useful insight for aisle organizations, sales, marketing campaigns, and more.
In this post, we will analyze the grocery dataset available on Kaggle. Let’s start with reading the dataset.
import numpy as npimport pandas as pdgroceries = pd.read_csv("/content/Groceries_dataset.csv")groceries.shape(38765, 3)groceries.head()
The dataset is organized in a way that each row represents an item purchased on a given day by a particular customer.
Before starting on the analysis, we should check the data types, and if there are any missing values.
groceries.isna().sum().sum()0groceries.dtypesMember_number       int64 Date               object itemDescription    object
There is no missing value but the data type of “Date” column should be converted to datetime which can be done with the to_datetime function of pandas.
groceries.Date = pd.to_datetime(groceries.Date)
Let’s first check the average number of items sold per day. One way to do this is group items by date and count the items. We can then plot the result.
import matplotlib.pyplot as pltgroceries[['Date','itemDescription']].groupby('Date').count()\.plot(figsize=(12,6), legend=False, fontsize=14)plt.title('Number of Items Sold per Day', fontsize=18)plt.xlabel('Date',fontsize=14)plt.ylabel('Qty', fontsize=14)
It’d look better if we downsample it. We can use the resample function to decrease the frequency.
groceries[['Date','itemDescription']].groupby('Date').count()\.resample('M').mean()\.plot(figsize=(12,6), legend=False, fontsize=14)plt.title('Number of Items Sold per Day', fontsize=18)plt.xlabel('Date',fontsize=14)plt.ylabel('Qty', fontsize=14)
It seems like business is getting better because item quantities follow a general increasing trend.
Each row in the dataset represents an item purchased by a customer on a given day. If a customer buys three items at one shopping, there will be three rows with same customer number and date but with different item description.
Another measure is the average number of items per shopping which can be calculated by grouping the items by customer number and date.
item_qty = groceries[['Member_number', 'Date','itemDescription']]\.groupby(['Member_number','Date']).count().reset_index()item_qty.head()
Item_qty dataframe shows the number of items in each shopping. For instance, customer 1000 purchased 3 items on 2014–06–24.
The average number of items per shopping is around 2.5 .
item_qty.itemDescription.mean()2.5907
Let’s also check the distribution of the number of items per shopping.
item_qty.itemDescription.plot(figsize=(10,6), kind='hist',                              legend=False, fontsize=14)plt.title('Histogram of Item Quantities per Shopping', fontsize=18)
Customers are most likely to buy 2–3 items together.
The main focus of market basket analysis is what items are purchased together. One common technique is association rule learning which is a machine learning method to discover relationships among variables. Apriori algorithm is a frequently used algorithm for association rule learning.
We will not go in detail about apriori algorithm or association rule learning in this post. Instead, I will show you a simple way to check which items are frequent purchased together.
Let’s first create a dataframe that contains the items list per shopping.
items = groceries.groupby(['Member_number', 'Date'])\.agg({'itemDescription': lambda x: x.ravel().tolist()}).reset_index()items.head()
Three items purchased by customer 1000 on 2014–06–24 are whole milk, pastry, and salty snack.
We need to determine which items frequently exist in the same rows in “itemDescription” column.
One way is to create combinations of items in each row and count the occurrences of each combination. The itertools of python can be used to accomplish this task.
Here is an example for the first row.
import itertoolslist(itertools.combinations(items.itemDescription[0], 2))
There are 3 items in the first row so we have 3 combinations of pairs. Itertools.combinations do not return repeated combinations (e.g. (‘pastry’, ‘pastry’)) which is what we need.
The following code will to this operation on each row and add the combinations to a list.
combinations_list = []for row in items.itemDescription:    combinations = list(itertools.combinations(row, 2))    combinations_list.append(combinations)
We have created a list of lists:
We can create a list from this list of lists by using the explode function of pandas but we need to first convert it to a pandas series.
combination_counts = pd.Series(combinations_list).explode().reset_index(drop=True)
We can now count the number of occurrences of each combination using the value_counts function. Here are the ten most frequent combinations:
The first one is a surprise because it is repeating one. We have made the combinations with no repeated elements. The dataset might contain repating elements. For instance, if a customer buys 2 whole milks at one shopping, there must be two rows of whole milk for that shopping.
We can confirm it by counting the number of whole milks at each shopping.
whole_milk = groceries[groceries.itemDescription == 'whole milk']\.groupby(['Member_number','Date']).count()\.sort_values(by='itemDescription', ascending=False).reset_index()whole_milk.head()
It seems like what we suspect is correct. For instance, customer 1994 purchased 4 whole milks on 2015–11–03. Thus, it totally makes sense to see the most frequent combination is whole milk and whole milk.
We should focus on non-repeating combinations. For instance, the second most frequent combination is whole milk and rolls/buns. The whole milk seems to be dominating the shopping lists.
We have done a simple and basic market basket analysis. The big retailers are doing much more complicated ones and they may approach the analysis from many different perspectives.
However, the overall goal is usually the same which is to be able to predict customers purchasing behaviour.
Thank you for reading. Please let me know if you have any feedback.