Customer Segmentation with RFM analysis — Part 3
By: Aman Prasad
Tags: Machine Learning, Clustering, Feature Engineering, Data Science

In this post we develop some extensions to further improve our customer segmentation model.

As analysts/data scientists/research or applied scientists we are always looking for ways to improve the performance of our models. It is important to always look at the context of the improvement that we need. In part 2 of this series we speculated that in the future we might need to consider the following for improving the model:

Adding more variables to the feature set

Alternative ML algorithms

Use a different clustering technique

Use a larger data source

Let us walk through some practical methods to improve our prediction model.

Before we start, I want to mention that I am using the same data source, so we can rule out the point ‘use a larger data source’ and facilitate comparison to the previous methods.

Adding more variables to the feature set

Adding more features to our model (especially when we are using few predictors) is always a good idea. To add variables we need to consider the availability of data and the context of our response or target variable. Since we are predicting the revenue segments of customers, we need to ask ourselves — what could be some factors that drive revenue? Which factors could help us distinguish customers belonging to different revenue segments?

Assuming one is using transaction data for building a segmentation model, a variable that might be readily available and will help improve predictive performance is what I call ‘UniqueItems’ (described by Khajvand et al., 2011). ‘UniqueItems’ is the number of distinct products a customer has bought. Our hypothesis is that the more diverse range of products a customer buys, the more dependent the customer is on the business and by extension will contribute to revenue more. As our RFM variables are on a customer level (that is, we have RFM scores for each customer), we need to calculate UniqueItems on the same level. With transaction data that looks like this:

A new column ‘ItemNumber’

We can easily calculate UniqueItems for each customer by taking the sum of distinct items that each customer has bought.

How do we know if our new derived is useful for prediction? In traditional regression based methods one might look for ‘p-values’, which if below a threshold (such as 0.05) tell us that the feature is statistically significant in explaining the target variable. With modern machine learning algorithms such as XGBoost, we can plot a variable importance plot, which looks something like this:

Feature importance plot with F score from XGBoost. ‘Item’ refers to UniqueItems

Before we go into feature importance, just a small detour: XGBoost is a technique which builds upon multiple decision trees. You might recall that in a decision tree algorithm such as CART, a tree is formed with branches splitting on the values of some features. CART minimizes the objective function by growing branches greedily from top to bottom (maximising Information Gain or minimizing GINI impurity at each level). Hence, CART chooses a split according to the objective function while considering other input variables and split points.

How is this related to feature importance? Well, lets say that XGBoost is REALLY careful while selecting features. XGBoost has a built-in mechanism which updates the model after every iteration, by fitting a new model to the residuals of the previous prediction.

In the above figure, X can be viewed as the inputs to the model. θ is the function space (or hypothesis space/model class) which contains different functions with different parameters. XGBoost will find a function f which minimizes the objective function considering the inputs X and the function space θ.

Considering this mechanism, it is safe to say that the features that XGBoost ends up using are actually useful to an extent and are not just noise. ‘F score’ from the feature importance plot tells us how many times a feature was used to make a split in the trees. As we can see, frequency seems to be the most important feature, with UniqueItems helping predictive performance as well. In case one is using a combinations of ML algorithms as an ensemble model, plotting the feature importance might be tricky. However, we can use XGBoost to tell us which features are important and proceed to use an unrelated ensemble model for prediction.

Great! We have successfully added a new feature to the model. Using the principles described above, we can add as many features as we can think of.

Use a different clustering technique

Previously in this series, we used k-means clustering to group our data. k-means has proven to be one of the most popular and simple clustering algorithms and is still widely used despite the fact that thousands of clustering algorithms have been published since (Jain, 2010). Jain (2010) also discuss how clustering remains a difficult problem, with one of the important questions when deciding the representation of data being the purpose of grouping. The representation of data should be chosen according to the end goal of the model.

In our case, one of the issues we came across was that when we cluster our revenue or LTV variables, the distribution of customers in the segments was a tad untidy. For both Companies, we have too few customers in higher value segments, and too many in low value segment. Since we are predicting revenue segments to tell managers which customers to focus on, it would be better to have more customers in higher value segments. We then will be able to give managers more options to choose from.

LTV segments distribution using k-means clustering

To solve this problem, we can look to other clustering techniques such as Gaussian mixture modelling (GMM). A good article which summarizes the math behind GMM can be found here.

Let us talk about some limitations of k-means and how GMM helps us overcome them. k-means performs ‘hard assignment’, which implies only points inside the circlular cluster are part of the cluster and any point outside is not. GMM (Gaussian Mixture Models) instead gives soft boundaries, where each data point is assigned a probability of belonging to each cluster. For example, with k=2, a datapoint can belong to a cluster with 80% probability and belong to the other cluster with 20% probability. This could be a desirable property in the sense it provides more information than just a cluster assignment. Another advantage could be detecting outliers (which k-means is heavily affected by): points with a lower probability of belonging to a cluster (for example: a point has probability 0.5 to belonging to a cluster with k=2) than some determined threshold could considered as outliers (one could set a rule such as the probability of belonging to the assigned cluster should higher than some threshold).

k-means lacks flexibility in cluster shape as it tries to fit circles (in higher dimensions, a hypersphere) to cluster the data. k-means hence assumes that the clusters must be circular and cannot account for other shapes. Given actual cluster shapes in the data which are not circular, k-means will still fit circular clusters and may result in poor clustering. Considering this GMM can be viewed as an attractive alternative, which fits ellipses rather than circles to account for non-circular clusters.

We can try to visualize k-means clustering on the 3-dimensional RFM data by projecting them into 2 dimensions using Principal Component Analysis (PCA). In the below plots, we first scale the RFM data, project it into 2D, then run k-means clustering (k=3 for Company A, k=4 for Company B). In the below plots the grey circles represent how k-means fits circles with the radius defined as the distance to the most distant point in the cluster.