Instacart Users Segmentation and Market Basket Analysis
By: Yingyuan, Valerie
Tags: Instacart, Machine Learning, Data Science, Data Visualization

Instacart Users Segmentation and Market Basket Analysis

Understanding the customer shopping behaviors of Instacart and make efficient recommendation

Written by: Yingyuan (Valerie) Zhang, Yutong (Grace) Zhu Zehui (Declan) Xiang, Xin Yu

Covid-19 has been a world spread pandemic in 2020. Thus, New Yorkers follow the quarantine policy and keep a social distance from each other. As we know, the most popular mode of transportation in NYC is by subway. However, the subway is one of the most dangerous transportation tools to spread Covid-19, which may increase the risk of having Covid-19 for subway passengers. Therefore, going out to get daily needs became a headache issue for New Yorkers. People in grocery stores are not practical to keep a social distance. During Covid-19 outbreaks, New York City published a “staying at home” order, which increased the demand for online grocery shopping. Instacart is a grocery delivery platform that has experienced rapid growth during the Covid-19 crisis. Now, users gain the value of staying home to flatten the curve and to reduce their own risk of getting the virus.

The primary research goals are doing user segments based on time intervals and building a recommendation system based on product choices of users. The expectation of the research could optimize the Supply side’s inventory allocation and increase the probability that customers get essential goods without breaking the social distancing rule.

First, let’s explore the data!

The primary data source is from Instacart’s 2017 anonymized customers’ orders over time (Stanley, 2017). It contains the order file, product file, order and product file, aisles file, and department file. Each entity in the dataset has an associated unique id.

In the order dataset, it contains user id, order id, order purchased day of the week(order_dow), order purchased hour of the day(order_hour_of_the_day), days since the last purchase(day_since_prior) and an indicator of the order’s belongs(eval_set). If it is a first time purchase, the days since the last purchase will be NaN. In the department dataset, it contains an unique department id and associated departments’ names. In the aisles dataset, it has aisle id and aisles’ names. In the product dataset, it contains the product id, the name of the product, the aisles’ id and the department id.

To make the time interval of user orders, we first divided user orders based on days. The data we used here is order.csv, column name ‘dow’. From Figure 1, The most popular days of user orders are days 0 and 1. After reviewing the data instruction, we did not find the definition of days 0 to 6. We believe the two busy days, 0 and 1, should be Sunday and Monday.

Figure 1 (Image by Author)

Having a more detailed view, we selected the hour as a unit to plot an order frequency bar chart. The two tails indicate that Instacart has not much delivery in the morning and late evening time. Most orders are placed in time interval hours [9,16], which has an average of around 25000 orders per hour.

Figure 2 (Image by Author)

Combining the two figures above, we decided to use a heatmap to check the most popular order time. The y-axis is order counted by days, and the x-axis is order counted by hours. From Figure 3, the darker part means the higher density time slots of having user orders, which are around day 0 and day 1 from 9 a.m to 4 p.m.

Figure 3 (Image by Author)

Figure 4 shows a decreasing trend. Most of the users have placed orders in the frequency interval [0,40]. A few users have orders above 60. Unique orders count above 100 have been counted as a group. The x-axis has a limitation on putting all the count orders on one chart.

Figure 4 (Image by Author)

An interesting finding, from Figure 5, We have found that over 400,000 users may add Banana to their orders. Most of the products on the list are organic vegetables and fruits!

Figure 5 (Image by Author)

Machine Learning Modeling:

K-Means:

Built on EDA results, it is likely to conclude that consumers are more likely to purchase between 10 am to 4 pm. Also, it is shown by the graph on day 0 and day 1, Instacart has a relatively high volume compared with other days. To further understand consumer’s purchase behavior and conduct a concrete consumer segment, we used the k-means clustering. Not only based on the order purchased the day of the week, the hour of the week but also days since prior purchase and add to cart order, to calculate the frequency of using Instacart and median of items bought per order for each user.

First, selected the optimal number of clusters in k-means using the Silhouette Score.

Due to the processing volume, we sampled 5% of the dataset (more than 10,000 user records) five times to generate the accurate Silhouette score for the train dataset. As it suggested, we set 3 clusters in the k-means algorithm. After applying PCA to lower the dimension, we were able to plot each cluster (Fig. 6).

Figure 6 (Image by Author)

Random Forest:

After doing k-means clustering and labeled the data, which we used to train our first K-means model with cluster 0,1,2. We want to transfer our model from unsupervised learning to supervised learning. Thus, we split our dataset into train and test samples. By using GridSearchCV, we find that we should use 21 as our estimator and 97 as our max leaf node. After training our model and fit the test set, we calculated the multi-class ROC AUC score, which is 0.999, which means our random forest model separates the three classes very well.

Feature Importance (Image by Author)

Recommendation System

In our recommendation system model, the outputs are departments and aisles that most other customers would purchase but the input customers do not purchase often. Since the outputs are not only based on the majority of trained data but also based on Pearson similarity of the inputs customers purchase history, our model considers different groups of customer preferences.

For time and computing saving reasons, we used data from the first 10,000 users to train our model. By taking 5 random users from each cluster as input, the results have proved our hypothesis. For 5 users from each cluster and 5 recommended departments and aisles, we look for the top 3 most shared common names among 25 values.

Departments recommended:

Cluster 0: alcohol, bulk, babies

Cluster 1: alcohol, dry food and pasta, bulk

Cluster 2: alcohol, bulk, meat and seafood

Aisles recommended:

Cluster 0: facial care, skin care, hair care

Cluster 1: bulk grains rice dried goods, facial care, skin care

Cluster 2: skin care, spirits, facial care

From this table, we can tell that customers in cluster 0 and cluster 1 (64.6% of total users) are loyal Instacart users and they might have high demand and put heavy pressures on Instacart’s operation during Covid-19.