RFMT Segmentation Using K-Means Clustering | by Yexi Yuan | Towards Data Science

Important note: This was created as part of my own personal learning process for data science in python. I find it extremely helpful when i write this down to help me learn better and faster. This was part of the course on DataCamp and the code is based on the course and other online resources I used. Please visit DataCamp for the original syllabus. I am not affiliated with DataCamp in any form and only use their resources as a learner.
On the final part of our customer segmentation journey we will be applying K-Means clustering method to segment our customer data. We will continue to use the features we’ve engineered in our RFM model. Additionally, we will be including tenure as a new feature for our model to create RFMT model.
K-Means is a popular and simple unsupervised machine learning algorithm. Put simply, the K-means algorithm identifies k number of centroids, and then allocates every data point to the nearest cluster, while keeping the centroids as small as possible. We will not dive into the specifics of how K-Means work so let us dive into the customer segmentation implementation.
These will be the 4 features that we are going to use to implement our K-Means model. Before we can fit the K-Means model to our data, we need to ensure that these key assumptions are fulfilled.
Distribution of variablesVariables with same average valuesVariables with same variance
Distribution of variables
Variables with same average values
Variables with same variance
Looking at our RFMT summary statistics, we can see that our data set do not fulfill these key assumptions at the moment. Let’s create a data pre-processing pipeline to prep our data to be K-Means ready.
Below are the steps we need to take to perform our data transformation and dive into each step in details:
Unskew the data — We will be using log transformation to do thisStandardize to the same average valuesScale to the same standard deviationStore as a separate array to be used for clustering
Unskew the data — We will be using log transformation to do this
Standardize to the same average values
Scale to the same standard deviation
Store as a separate array to be used for clustering
Let’s examine the shape of the distribution of our current data set.
# Plot RFM distributionsplt.figure(figsize=(16,14))# Plot distribution of Rplt.subplot(4, 1, 1); sns.distplot(data_process['Recency'])# Plot distribution of Fplt.subplot(4, 1, 2); sns.distplot(data_process['Frequency'])# Plot distribution of Mplt.subplot(4, 1, 3); sns.distplot(data_process['MonetaryValue'])# Plot distribution of Tplt.subplot(4, 1, 4); sns.distplot(data_process['Tenure'])# Show the plotplt.show()
As you can see, these is a general skewness of R, F, and M to the right. T has a more evenly distributed shape. To address this, we will apply Log Transformation to our data.
# Apply Log Transformationdata_process['MonetaryValue'] = data_process['MonetaryValue'] + 0.0000000001recency_log = np.log(data_process['Recency'])frequency_log = np.log(data_process['Frequency'])monetary_log = np.log(data_process['MonetaryValue'])tenure_log = np.log(data_process['Tenure'])# Plot RFM distributionsplt.figure(figsize=(16,14))# Plot distribution of Rplt.subplot(4, 1, 1); sns.distplot(recency_log)# Plot distribution of Fplt.subplot(4, 1, 2); sns.distplot(frequency_log)# Plot distribution of Mplt.subplot(4, 1, 3); sns.distplot(monetary_log)# Plot distribution of Mplt.subplot(4, 1, 4); sns.distplot(tenure_log)# Show the plotplt.show()
Great! Looks like we have normalized our RFM features. However, notice that by log transforming T, we’ve caused the data to left-skew. We have to always be careful when transforming data by examine through visualization or otherwise to ensure we create the most accurate representation of our customer segments.
In order to standard mean and standard deviation, we can use the following formula:
datamart_centered = datamart_rfm - datamart_rfm.mean()datamart_scaled = datamart_rfm / datamart_rfm.std()
However, let’s make use of SKLearn library’s StandardScaler, to center and scale our data instead.
from sklearn.preprocessing import StandardScalerscaler = StandardScaler()scaler.fit(data_process_log)data_process_norm = scaler.transform(data_process_log)data_process_norm_df = pd.DataFrame(data_process_norm)data_process_norm_df.describe().round(2)
Awesome! Now we have fulfilled the key assumptions in order for us to properly and accurately apply K-Means clustering to segment our customers.
Now let’s get into the most exciting portion of any machine learning project — applying the ML model!
But before that, we need to choose the number of clusters for our K-Means model. This can be done in a few ways:
Visualization Method — The Elbow CriterionMathematical Method — Silhouette CoefficientExperimentation and interpretation that makes the most business sense
Visualization Method — The Elbow Criterion
Mathematical Method — Silhouette Coefficient
Experimentation and interpretation that makes the most business sense
We will be using the elbow criterion method.
The sum-of-squared-errors (SSE) is the sum of squared distances from every data point to their cluster center. We want to choose the optimal number of clusters that reduces the number SSE without over-fitting. Let’s plot our elbow chart to determine that.
# Fit KMeans and calculate SSE for each *k*sse = {}for k in range(1, 11):    kmeans = KMeans(n_clusters=k, random_state=1)    kmeans.fit(data_process_norm)    sse[k] = kmeans.inertia_# Plot SSE for each *k*plt.title('The Elbow Method')plt.xlabel('k'); plt.ylabel('SSE')sns.pointplot(x=list(sse.keys()), y=list(sse.values()))plt.show()
Ideally, we want to choose the point on the elbow chart where the SSE stops decreasing at an increasing rate — i.e. the point where the change gradient of between the number of clusters becomes constant. For our model we will choose k=4.
# Import KMeans from skLearnfrom sklearn.cluster import KMeans# Choose k=4 and fit data set to k-means modelkmeans = KMeans(n_clusters=4, random_state=1)kmeans.fit(data_process_norm)# Assign k-means labels to cluster labelscluster_labels = kmeans.labels_# Assign cluster labels to original pre-transformed data setdata_process_k4 = data_process.assign(Cluster = cluster_labels)# Group data set by k-means clusterdata_process_k4.groupby(['Cluster']).agg({    'Recency': 'mean',    'Frequency': 'mean',    'MonetaryValue': 'mean',    'Tenure': ['mean', 'count']}).round(0)
We can see that our grouped summary of the mean of R, F, M, and T that each cluster of customers places a different emphasis on our 4 features:
Cluster 0 has the highest MontaryValue mean and lowest Recency mean and the highest frequency mean — This is our ideal customer segment
Cluster 1 performs poorly across R, F, and M and has a long tenure in our database as well — we will need to design campaigns to activate them again
Cluster 2 shopped with us recently but have not spend as much or as frequently as we would like them to — perhaps some personalization of products targeted at them can help to maximize their lifetime-value?
Cluster 3 has spent quite a fair amount with us but has not shopped with us in the 3–4 months — We will need to do something before we lose them!
We can visualize the relative importance of each feature for our 4 clusters through a heatmap.
# Calculate average RFM values for each clustercluster_avg = data_process_k4.groupby(['Cluster']).mean()# Calculate average RFM values for the total customer populationpopulation_avg = data_process.mean()# Calculate relative importance of cluster's attribute value compared to populationrelative_imp = cluster_avg / population_avg - 1# Initialize a plot with a figure size of 8 by 2 inches plt.figure(figsize=(8, 4))# Add the plot titleplt.title('Relative importance of attributes')# Plot the heatmapsns.heatmap(data=relative_imp, annot=True, fmt='.2f', cmap='RdYlGn')plt.show()
The heatmap provides a simple and easy way to understand how our K-Means model place relative importance of our RFMT attributes to assign each customer to their respective segments.
Lastly, let’s visualize our K-Means Clusters in scatter plots to warp up our customer segmentation journey.
# Plot RFM distributionsplt.figure(figsize=(20,20))plt.subplot(3, 1, 1);plt.scatter(data_process_k4[cluster_labels == 0].loc[:,'Recency'], data_process_k4[cluster_labels == 0].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 1', alpha=0.2)plt.scatter(data_process_k4[cluster_labels == 1].loc[:,'Recency'], data_process_k4[cluster_labels == 1].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 2', alpha=0.2)plt.scatter(data_process_k4[cluster_labels == 2].loc[:,'Recency'], data_process_k4[cluster_labels == 2].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 3', alpha=0.3)plt.scatter(data_process_k4[cluster_labels == 3].loc[:,'Recency'], data_process_k4[cluster_labels == 3].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 4', alpha=0.3)plt.xticks(np.arange(0, 1000, 100)) plt.yticks(np.arange(0, 10000, 1000))axes = plt.gca()axes.set_xlim(0, 500)axes.set_ylim(0, 10000)plt.title('Cusomter Clusters')plt.xlabel('Recency')plt.ylabel('Monetary Value')plt.legend()plt.subplot(3, 1, 2);plt.scatter(data_process_k4[cluster_labels == 0].loc[:,'Frequency'], data_process_k4[cluster_labels == 0].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 1', alpha=0.2)plt.scatter(data_process_k4[cluster_labels == 1].loc[:,'Frequency'], data_process_k4[cluster_labels == 1].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 2', alpha=0.2)plt.scatter(data_process_k4[cluster_labels == 2].loc[:,'Frequency'], data_process_k4[cluster_labels == 2].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 3', alpha=0.3)plt.scatter(data_process_k4[cluster_labels == 3].loc[:,'Frequency'], data_process_k4[cluster_labels == 3].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 4', alpha=0.3)plt.xticks(np.arange(0, 1000, 100)) plt.yticks(np.arange(0, 10000, 1000))axes = plt.gca()axes.set_xlim(0, 500)axes.set_ylim(0, 10000)plt.title('Cusomter Clusters')plt.xlabel('Frequency')plt.ylabel('Monetary Value')plt.legend()plt.subplot(3, 1, 3);plt.scatter(data_process_k4[cluster_labels == 0].loc[:,'Tenure'], data_process_k4[cluster_labels == 0].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 1', alpha=0.2)plt.scatter(data_process_k4[cluster_labels == 1].loc[:,'Tenure'], data_process_k4[cluster_labels == 1].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 2', alpha=0.2)plt.scatter(data_process_k4[cluster_labels == 2].loc[:,'Tenure'], data_process_k4[cluster_labels == 2].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 3', alpha=0.3)plt.scatter(data_process_k4[cluster_labels == 3].loc[:,'Tenure'], data_process_k4[cluster_labels == 3].loc[:,'MonetaryValue'], s= 10, cmap='rainbow', label = 'Cluster 4', alpha=0.3)plt.xticks(np.arange(0, 1000, 100)) plt.yticks(np.arange(0, 10000, 1000))axes = plt.gca()axes.set_xlim(0, 500)axes.set_ylim(0, 10000)plt.title('Cusomter Clusters')plt.xlabel('Tenure')plt.ylabel('Monetary Value')plt.legend()plt.show()
Thanks for taking time to explore customer segmentation with through this 3-part series. If you enjoyed the content, stay tuned and I will be using exploring more data and perhaps build other interesting projects as well with Python.