Instacart Market Basket Analysis Part 2: Which Groups of Customers Are Similar? | by James Le | Towards Data Science

The data science environment is rapidly changing and getting more mature. Whether you’re an experienced data scientist building developing cutting-edge deep learning algorithms or you’re a graduate student grinding on school projects, being able to publish and collaborate on your Jupyter Notebooks (the default workbench in data science) with almost no effort is quite critical.
Thus far, there have not been many simple solutions to share notebooks that your team can collaborate on. If you’re an entry-level data scientist fresh out of school, this can be a big problem. I recently got introduced to Saturn Cloud and found that it is super simple to share work and use cloud-based computing.
This is part 2 of a 3-part article series where I give an example of how I use Saturn Cloud to communicate my results working with the Instacart Market Basket Analysis challenge. In part 1, I did the exploratory data analysis phase to understand in fine-grained details about the customer shopping behavior on the Instacart platform.
In part 2, I will try to find a possible customer segmentation enabling us to classify customers according to their different purchases. Customer segmentation is the sub-division of a customer base into discrete groups that share similar characteristics. This method can be a powerful way to identify unsatisfied customer needs. Using this information, Instacart can then outperform its competition by developing uniquely appealing products and services. Furthermore, this information will also be useful for the next prediction task that can be used to recommend new items to the customers.
Since there are thousands of products in the dataset, I will rely on aisles, which represent categories of products. Even with aisles, the number of features will still be too much. Thus, I will use Principal Component Analysis to find new dimensions along which clustering will be easier and then try to find possible explanations for the identified clusters.
Continuing from the data exploration in part 1, this is my understanding of the dataset structure:
Users are identified by user_id in the orders.csv file. Each row of the orders.csv file represents an order made by a user. Orders are identified by their order_id.
Each order of a user is characterized by an order_number which specifies when it has been made with respect to the others of the same user.
Each order consists of a set of products, where each characterized by an add_to_cart_order feature representing the sequence in which they have been added to the cart in that order.
For each user, we may have (n — 1) prior orders and 1 train order OR (n — 1) prior orders and 1 test order in which we have to state what products have been reordered.
To make it easier for the analysis, I merged all the different data frames into a single one called mt.
_mt = pd.merge(prior,products, on = ['product_id','product_id'])_mt = pd.merge(_mt,orders,on=['order_id','order_id'])mt = pd.merge(_mt,aisles,on=['aisle_id','aisle_id'])
We are dealing with 134 types of aisles and 49,677 different products. As indicated to the left, the top 5 best selling goods are fresh fruits, fresh vegetables, packaged vegetable fruits, yogurt, and packaged cheese.
The goal is to find all possible clusters among the different customers and substitute the single user_id with the cluster to which they are assumed to belong. This would eventually increase the next prediction model performance.
The first thing to do is creating a data frame with all the purchases made by each user. This cust_prod has 206,209 users and 134 columns.
cust_prod = pd.crosstab(mt['user_id'], mt['aisle'])cust_prod.head(10)
I then apply the Principal Component Analysis algorithm to the cust_prod data frame. This helps reduce the number of features from the number of aisles to 6, which is the number of principal components I have chosen.
Saturn Cloud really helps here because of the computationally expensive nature of doing the PCA algorithm. Since I hosted my notebook on their cloud-enabled platform, I can take advantage of the GPU that Saturn provides to execute this expensive operation.
from sklearn.decomposition import PCApca = PCA(n_components=6)pca.fit(cust_prod)pca_samples = pca.transform(cust_prod)ps = pd.DataFrame(pca_samples)ps.head()
After plotting several pairs of components and looking for the one suitable pair for a K-Means Clustering, I have chosen the (PC4, PC1) pair as seen below. Since each component is the projection of all the points of the original dataset, I think each component is well representative of the dataset.
Here’s how our clusters appear. You can see that there are 4 clusters associated with 4 different colors (blue, green, purple, and orange). The value 0, 1, 2, 3 circles are the centroids of each cluster.
I think I have found a possible clustering for Instacart customers. Let’s see what are the top 10 goods bought by people from each cluster. I will rely first on the absolute data and then on a percentage among the top 8 products for each cluster.
Below are the top 10 product goods in cluster 0.
c0.sort_values(ascending=False)[0:10]
Below are the top 10 product goods in cluster 1.
c1.sort_values(ascending=False)[0:10]
Below are the top 10 product goods in cluster 2.
c2.sort_values(ascending=False)[0:10]
Below are the top 10 product goods in cluster 3.
c3.sort_values(ascending=False)[0:10]
A glimpse analysis of the clusters confirm the initial hypothesis that (1) fresh fruits, (2) fresh vegetables, (3) packaged vegetable fruits, (4) yogurt, (5) packaged cheese, (6) milk, (7) water seltzer sparkling water, and (8) chips pretzels are products which are generically bought by most of the customers.
What I can inspect here is whether or not the clusters differ in quantities and proportions with respect to these goods, or if a cluster is characterized by some goods not included in this list. For instance, I can already see cluster 3 is characterized by ‘Baby Food Formula’ product, which is a significant difference with respect to the other clusters.
The following table depicts the percentage of these goods with respect to the other top 8 in each cluster. It is easy some interesting differences among the clusters.
It seems people of cluster 1 buy more fresh vegetables than the other clusters. As shown by absolute data, Cluster 1 is also the cluster including those customers buying far more goods than any others.
People of cluster 2 buy more yogurt than people of the other clusters.
People of cluster 3 buy a Lot of ‘Baby Food Formula’ which not even listed in the top 8 products but mainly characterizes this cluster. Coherently (I think) with this observation they buy more milk than the others.
I think another interesting information might come from looking at the 10th to 15th most bought products for each cluster which will not include the generic products (i.e. vegetables, fruits, water, etc.) bought by anyone.
In cluster 0, the 10th to 15th most bought goods are refrigerated, ice cream, frozen produce, eggs, and crackers.
c0.sort_values(ascending=False)[10:15]
In cluster 1, the 10th to 15th most bought goods are soy lactose-free, bread, crackers, cereal, and candy chocolate.
c1.sort_values(ascending=False)[10:15]
In cluster 2, the 10th to 15th most bought goods are eggs, canned jarred vegetables, bread, chips pretzels, and refrigerated.
c2.sort_values(ascending=False)[10:15]
In cluster 3, the 10th to 15th most bought goods are soy lactose-free, frozen produce, water seltzer sparkling water, refrigerated, and eggs.
c3.sort_values(ascending=False)[10:15]
As you can note by taking into account more products, the clusters start to differ significantly.
For any business, doing customer segmentation has a variety of advantages: determining the appropriate product pricing, developing effective and customized marketing campaigns, designing an optimal distribution strategy, choosing specific product features for deployment, and prioritizing new product development efforts. As evidenced in my analysis, there are a variety of ways that Instacart can execute any of the above business strategies with the large customer base that the company has.
If you are interested in reproducing this result yourself, you can download the dataset from the Instacart website and view my notebook on Saturn Cloud here. In the final part of this blog post series, I will attempt to do association rule mining, a machine learning technique that helps to show the probability of relationships between products purchased within the Instacart orders dataset.