Clustering Algorithm for Customer Segmentation
By: Destin Gong
Tags: K Means, Machine Learning Python, Clustering Analysis, Data Science, Scikit Learn

If you would like more details about data visualization and EDA, please check out these two articles.

2. Data Preprocessing

Now that we define the objectives and have a better understanding of our data, we need to preprocess the data to meet the model requirement. In this example, I am choosing K Means clustering as the main algorithm and the reason will be explained later on.

The standard k means algorithm is not directly applicable to categorical data. To be more specific, the search space of categorical variables (e.g. Gender in this example) is discrete (either male or female), hence cannot be directly combined with a continuous space and measured the distance in the same manner. Therefore I discarded the “Gender” field as well as “CustomerID” field.

Since K means interpret the closeness between data points based on Euclidean distance, it is important to reconcile all dimensions into a standard scale. An appropriate type of data transformation should be selected to align with the distribution of the data. This article from Google provides a general guidance of data transformation in clustering.

In Summary:

if normal distribution → normalization

if power law distribution → log transformation

if data doesn’t conform to any distribution → quantiles

From the earlier univariate analysis, we can see that those variables don’t conform to either normal distribution or law distribution. Therefore, I use MinMaxScaler to shrink the data range to between 0 and 1, while maintaining the distribution shape.

data transformation (image by author)

3. EDA

Exploratory Data Analysis provides visual clues about whether it is likely to form insightful clusters when combining multiple variables together. It is also an imperative step because choosing an appropriate clustering algorithm is reliant on the shape of the cluster. Some center-based algorithms (e.g. K Means) are more adaptable towards globular shapes clusters and they tend to break linear shaped clusters apart. While density-based algorithms (e.g. DBSCAN) are better at clusters with irregular shape and a lot of noise.

I have visualized those three fields in the following way.

2D Scatter Plot

age vs. annual income

annual income vs. spending score

age vs. spending income

2D scatter plot (image by author)

3D Scatter Plot:

age vs. annual income vs. spending score

3D scatter plot (image by author)

In this case, it is apparent that the plot of “annual income vs. spending score” generates some center-based clusters. Therefore, I will use K means algorithm for further exploration.

K Means Clustering

1. How does K Means Clustering work?

K Means Clustering is a centre-based clustering algorithm, which means that it assigns data points to clusters based on closeness or distance, following these procedures:

Specify the number of clusters “K”. Initiate K random centroids and assign each cluster a centroid: Centroid is the center of each cluster. There are k data points randomly selected as the centroids at the beginning, and the cluster label of other data points are later defined relatively to them. Consequently, different initial centroid assignments may lead to different cluster formations. Form K clusters by assigning data points to the closest centroid: The closest centroid is usually defined by the smallest Euclidean distance but it can also be correlation or cosine depends on the use cases. Recompute the centroid of each cluster : After all data points have been assigned to a cluster, for each cluster we recalculate the mean of all data points belonging to that cluster and define it as the new centroid. Reach convergence when centroids no longer change: Iterate step 2–4 until reaching the stoping criteria that either the centroids no longer change or the maximum number of iterations are reached.

This procedure also determines that K Means algorithm has the limitation of clustering points into circular shapes with similar size and it is also heavily reliant on the predefined number K.

2. How to build a K Means Clustering model?

Now let’s take a closer look at how to implement it using python scikit learn.

Firstly, I define a KMeans_Algorithm function that pass the dataset and the number of clusters as the parameters.

define KMeans_Algorithm function (image by author)

This function calls the KMeans() from sklearn library and specify some key parameters:

n_clusters: the number of clusters

the number of clusters init: it controls initialization techniques and I chose “k-means++” to select initial centroids in a smart way to speed up convergence

it controls initialization techniques and I chose “k-means++” to select initial centroids in a smart way to speed up convergence max_iter: specify the maximum number of iterations allowed for each run

specify the maximum number of iterations allowed for each run algorithm : there are several options to choose from, “auto”, “full” and “elkan” and I chose “elkan” because it is a K Means algorithm variation that uses triangle inequality to make it more time efficient.

: there are several options to choose from, “auto”, “full” and “elkan” and I chose “elkan” because it is a K Means algorithm variation that uses triangle inequality to make it more time efficient. random_state: use an integer to determines the random generation of initial centroids

The output will generates following attributes and evaluation metrics:

cluster_centres_: returns an array of the centroid locations

returns an array of the centroid locations inertia_: sum of squared distances of data points to their closest centroid

sum of squared distances of data points to their closest centroid labels_: the cluster label assigned to each data point

the cluster label assigned to each data point silhouette_score: the distance between the data point to other data points in the same cluster compares to the data points in the nearest neighbour cluster

In this example, I am mainly interested in the formation of clusters with regards to “Spending Score”. Therefore, I used dataset X1, X2, X3 as specified below to feed the model.

Notice that, we no longer needs to partition the dataset for training and testing for unsupervised models.

Age vs. Spending Score

The code below used the dataset X1 — scaled Age vs. scaled Spending Score, and visualize how the clustering changes as the number of clusters increased from 2 to 10.

K Means for age vs. spending score (image by author)

The first part of the code iterates through the number of clusters and generate a scatter plot for each, with the red square representing the centroid. As shown below, the clusters change as the specified “k” value changes.

clustering result for X1 (image by author)

Afterwards, I evaluate the model performance based on two metrics: inertia and silhouette coefficient.

Since clustering is an unsupervised algorithm, we cannot directly assess the model quality based on discrepancies between the predicted results and the actual results. Therefore, the evaluation should be based on the principle of minimizing intra-cluster distance while maximizing the inter-cluster distance. There are two evaluation methods that determine the optimal number of clusters based on this principle.

1) Elbow method using inertia:

Inertia measures the sum of squared distances of samples to their closest cluster centroid. With the same number of cluster, smaller the inertia indicates better clusters. The elbow method determines the optimal number of clusters by looking at the turning point (“elbow” point) of the graph below, in this case is 6.

Elbow Method (image by author)

2) Silhouette coefficient:

Here is the Silhouette coefficient formula that makes it appear daunting:

But basically it translates into minimizing intra-cluster distance while maximizing the inter-cluster distance. a(i) is the average distance of data point i to other data points in the same cluster, and b(i) is the average distance of data point i to all points in the nearest neighbor cluster. The aim is to minimize a(i) and maximize b(i), therefore when the coefficient is closer to 1 indicates better clustering. In this example, both 2 and 6 are showing a peak in the score.

Silhouette Coefficient (image by author)

Based on the results from both metrics, 6 is most likely to be the optimal number of clusters when plotting “Age” against “Spending Score”. However, either from the score or the visualizations, it is still not convincing enough to say that age and spending score together form insightful clusters.

Annual Income vs. Spending Score

From the EDA process, we observed that there are several distinct clusters from the chart. Now, let’s take a closer look at whether the model can differentiate customers into distinct segments as expected. The code is similar to the one above but just changing the input dataset into X2 — scaled Annual Income vs. scaled Spending Score.