K-means Clustering for Customer Segmentations: A Practical Real-world Example
By: Shiyu Gong
Tags: Customer Segmentation, Unsupervised Learning, Clustering, Data Analysis, K Means

How many different types of customers does my business have?

This question frequently crosses business owners’ mind. And the concept behind it is customer segmentation.

First, what is customer segmentation?

Broadly speaking, the goal of customer segmentation is to divide customers into groups that share certain characteristics. It is crucial for a company to understand its customer behavior and categorize customers based on their user behavior. With good segmentation, companies could develop different and appropriate strategies for each group of customers accordingly and achieve the best effect such as targeting customers effectively and improving the customer experience etc.

Second, why use K-means clustering?

Customer segmentation is often performed using unsupervised, clustering techniques and one of the most commonly used techniques is K-means clustering.

The K-means clustering algorithm works by finding like groups based on Euclidean distance, a measure of distance or similarity, such that each point is as close to the center of its group as possible. Those groups could be used to determine which factors group members relate to.

K-means clustering for segmentation is popular at both academy and industry. For instance, López utilized K-means to segment electricity customers in order to provide electric utilities with a volume of information to enable them to establish different types of tariffs. Niknam also mentioned in his article that “a popular technique for clustering is based on k-means such that the data is partitioned into K clusters.”

Airbnb Use Case

Another impressive case of implementing k-means clustering algorithm is how Airbnb analyzes new UI taxonomy to identify natural clusters/groupings that can be used to inform future operating model encompassing, agent training, smart workflows and skilling/targeting introduced by Yashwanth K.

First and foremost, the data team of Airbnb analyzed UIs in new taxonomy that have >200 tickets in each UI which represented 80% of vol. Then, 16 different metrics were chosen to cluster UIs. Based on evaluation of clustering performance, 6 clusters for K-means clustering are optimal. Finally, they managed to characterize and plot ticket level descriptive metric stats by cluster, which allowed Airbnb to understand how these clusters can be associated with levels of complexity, emotion and urgency.

According to the result of the K-means clustering analysis, several actionable recommendations came up.

Improve education of hosts about simple self-solvable issues — specifically around setting up and managing their listing; Streamline and enhance workflow/agent skills to handle different groups of user issues to achieve potential NPS and efficiency gains; Execute strategy for training in line with target groups/specializations.

A Practical Real-world Example

After introducing the background of K-means clustering for customer segmentations, I would like to share my own experience of leveraging K-means clustering for solving a real-world business problem.

For my practicum project, our client is a SaaS company and it is at an important stage of transforming from one-time purchase to contract subscription. As a result, getting to know how customers are using the products and whether customers will renew their contracts are of great importance to my client.

Our team tried to segment users based on 80M behavioral records, leveraging PCA and K-means clustering in Python. Finally, we managed to create six personas that enabled our client to better segment customers for improved customer service.

The detailed steps and codes are as follows.

Firstly, choose metrics. We chose and calculated 6 different metrics for each product, which are median of usage days, median of session numbers, median of session length, fluctuation of usage days, fluctuation of session numbers, and fluctuation of session length respectively. We held the believe that these 6 metric could reflect user usage behavior from different perspectives such as usage frequency, usage quality and also usage stability.

Secondly, normalize data. Before clustering, we need to normalize the metrics so that distance metrics aren't dominated by variables with larger units.

from sklearn import preprocessing

scaler = preprocessing.MinMaxScaler()

a = PRODUCT1.index

PRODUCT1_scaler = scaler.fit_transform(PRODUCT1_scaler)

PRODUCT1_scaler = pd.DataFrame(PRODUCT1_scaler, columns=['median_days', 'median_session','median_time', 'fluctuation_day', 'fluctuation_session','fluctuation_time'])

PRODUCT1_scaler.set_index(a, inplace=True)

Thirdly, pre-specify k. One of the most important hyperparameters of K-means clustering, number of clusters k, needs to be pre-specified. This is sometimes defined by your problem statement, or sometimes something that needs to be optimized if clustering is being used as a descriptive tool.

To determine K, some possible techniques:

elbow method - look at percentage of variance explained at each K, select one such that adding more clusters doesn't much improve the model gap-statistic Silhouette Coefficient

Here we chose to use elbow method and the result showed that 6 clusters were optimal.

from sklearn.cluster import KMeans

existing_df_3d = PRODUCT1_scaler.copy() kmeans = KMeans(n_clusters=6)

clusters = kmeans.fit(existing_df_3d) existing_df_3d.head()

Fourthly, leverage PCA to visualize the clustering results. Given that we have total 6 metrics, we could not use one plot to show the 6-dimension at the same time. Thus, PCA is a good way to reduce dimension and show the clustering results in 3-dimension for interpretation.

The first three principle components were chosen, which explained 50.3%, 29.1% and 12.2% of total variance respectively.

import matplotlib.pyplot as plt

%matplotlib inline from sklearn.decomposition import PCA pca = PCA(n_components=3)

pca.fit(PRODUCT1_scaler)

print(pca.explained_variance_ratio_)

print(pca.explained_variance_)

existing_3d = pca.transform(PRODUCT1_scaler)

existing_3d.shape existing_df_3d = pd.DataFrame(existing_3d).loc[:,:2]

existing_df_3d.index = PRODUCT1.index

existing_df_3d.columns = ['PC1','PC2','PC3']

existing_df_3d.head()

## Visualization of Clustering results based on PCA

!pip uninstall -q -y plotly

!pip install -q plotly def enable_plotly_in_cell():

import IPython

from plotly.offline import init_notebook_mode

display(IPython.core.display.HTML('''

<script src="/static/components/requirejs/require.js"></script> '''))

init_notebook_mode(connected=False)





from plotly.offline import init_notebook_mode

enable_plotly_in_cell()

init_notebook_mode(connected=False) import plotly.offline as py

import plotly.graph_objs as go trace1 = go.Scatter3d(

x=existing_df_3d["PC1"],

y=existing_df_3d["PC2"],

z=existing_df_3d["PC3"],

mode='markers',

text=existing_df_3d["cluster"],

marker=dict(

size=2,

color=existing_df_3d["cluster"],

# set color to an array/list of desired values



colorscale=[[0.0,"indianred"],[0.2,"darkorange"],[0.4,"yellow"],[0.6,"mediumseagreen"],[0.8,"skyblue"],[1.0, "indigo"]],



opacity=0.8

)

)

#print(trace1)

data = [trace1]

layout = go.Layout(

showlegend=False,

title="K-MEANS CLUSTERING",

scene = dict(

xaxis = dict(title="PC1"),

yaxis = dict(title="PC2"),

zaxis = dict(title="PC3")

),

width=1000,

height=900

)

fig = go.Figure(data=data, layout=layout)

py.iplot(fig, filename='3d-scatter-colorscale')

Finally, combine your clustering results with your business. After we successfully segmented our customers into 6 groups, it was not the end of the analysis. Considering that K-Means clustering is an unsupervised machine learning algorithm, it is rather hard to evaluate the performance. As a result, we should go back and conduct descriptive analysis for the original metrics by new defined clusters to see whether customers within different clusters have different usage behavior.

Violin Plots of Cluster Attribute Performance

Based on the evaluation of cluster attribute performance, we managed to conclude that the clusters we got from K-means clustering contain customers with different usage behavior. And we could use those metrics to describe the clusters. For interpretation, we labelled them as Dormant, Erratic, Unstable, Stable, Heavy and Heavy+ respectively. For example, for customers within Dormant group, usage days, session numbers, and daily usage time are low. Also, such usage pattern is stable because the fluctuation is also low.

These six personas enabled our client to better segment customers for improved user behavior understanding and customer service, resulting in potentially 2M annual savings.

In the blog, we have went through the basic concepts of customer segmentation and K-means clustering, and a practical real-world example of how clustering algorithms could drive meaningful insights by generating customer segments.

What we should also know is that although K-means clustering is widely used, there is no single, correct way to perform customer segmentation. In reality, the optimal characteristics and analytic approach vary depending upon the business objective.