Unsupervised Learning with Weighted K-Means
By: Mallika Dey
Tags: K Means, Unsupervised Learning, Clustering, Data Science, Machine Learning

K-Means is an easy to understand and commonly used clustering algorithm. This unsupervised learning method starts by randomly defining k centroids or k Means. Then it generates clusters by iteratively running the following two tasks:

Each data point is assigned to a cluster in such a manner that it is closer to its own cluster center than any other cluster center. In other words, the Euclidean distance of each data point with the centroids is minimum for its own cluster

For each cluster, the new “cluster center”, which is the arithmetic mean of all the points belonging to the cluster gets calculated

These two tasks are repeated until one of the following happens -

The sum of squared Euclidean distances for all data points is minimized

Iterations reach a predefined limit

In a possible use case when all data points should not have equal importance, there is a caveat with this method : the clusters and the centroids are not representative of the data points.

Suppose you are trying to segment the customers of a retail store based on their organic and local product purchase behavior. Some customers have high spend share on both categories, some have low on both and the rest have a mixed propensity. You may use this information later to design a marketing campaign. You also want to understand what is the representative value of spend share for each customer group, so that you can evaluate the campaign performance.

A large number of customers have high spend share on these categories, but their overall spend and visit frequency are so low that they are less likely to respond to the campaign. The data points for these customers are going to generate biased clusters and centroid values. In such a scenario, you can use weighted K-Means and assign observational weight to each customer data point.

Here is a step by step guide to generate weighted K-Means clusters using Python 3

Step 1: Import all libraries and generate random samples for the exercise. In this case I am creating blobs for the spend share in a year in the two categories and a skewed distribution for the overall yearly spend of the customer

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns; sns.set() # for plot styling

from sklearn.cluster import KMeans

from pandas import DataFrame

import random

from sklearn.datasets.samples_generator import make_blobs

from scipy.stats import skewnorm X, y = make_blobs(n_samples=5000, centers=4, n_features=2, random_state=25)

df = DataFrame(dict(Pct_Spend_in_Organic=(X[:,0])+10, Pct_Spend_in_Local=(X[:,1])+10))

df['Total_Spend']=(skewnorm.rvs(1000,size=5000)*5000)+100

ax=df.reset_index().plot(x='index',y="Total_Spend",kind="hist")

Step 2: Visualize the input data in a scatter plot

plt.style.use('default')

x=np.array(df['Pct_Spend_in_Local'])

y=np.array(df['Pct_Spend_in_Organic']) plt.figure(figsize=(15, 10))

plt.scatter(x, y, s=5, cmap='viridis',c='orange',label='Spend in Organic Products') plt.title('Pct Spend in Local vs Organic Products',fontsize=18, fontweight='bold')

plt.xlabel('Pct Spend in Local',fontsize=15)

plt.ylabel('Pct Spend in Organic',fontsize=15)

Step 3:

Define K-Means with 1000 maximum iterations

Define an array ‘X’ with the input variables

Define an array ‘Y’ with the column ‘Total_Spend’ as the observational weights

kmeans = KMeans(n_clusters=5, random_state=0, max_iter=1000)

X = np.array(df.drop(['Total_Spend'], 1).astype(float))

Y = np.array(df['Total_Spend'].astype(float))

Step 4: Run the weighted k-means clustering and enter the ‘X’ array as the input and ‘Y’ array as sample weights. Generate the cluster levels for all data points

wt_kmeansclus = kmeans.fit(X,sample_weight = Y)

predicted_kmeans = kmeans.predict(X, sample_weight = Y)

Step 5: Visualize the clusters and the centroids in a scatter plot

plt.style.use('default')

plt.figure(figsize=(15, 10))

plt.scatter(X[:,0], X[:,1], c=wt_kmeansclus.labels_.astype(float),s=10,cmap='tab20b',marker='x')

plt.title('Customer Spend Local vs Organic - Weighted K-Means',fontsize=18, fontweight='bold')

plt.xlabel('Pct_Spend_in_Local',fontsize=15)

plt.ylabel('Pct_Spend_in_Organic',fontsize=15) centers = wt_kmeansclus.cluster_centers_

plt.scatter(centers[:, 0], centers[:, 1], c='black', s=500, alpha=0.5);

Step 6: Join the cluster labels and the centroids with the data-frame

df['ClusterID_wt'] = predicted_kmeans

centersdf=pd.DataFrame(centers)

centersdf['ClusterID_wt'] = centersdf.index

centersdf=centersdf.rename(columns = {0:'Wt Centroid: Spend in Local',1:'Wt Centroid: Spend in Organic'})

df=df.merge(centersdf,on='ClusterID_wt',how='left')

df.head()

Comparison: K-Means clustering with and without the observational weights

Run the clustering exercise, without the observational weights and join the cluster labels and centroids with the data frame.

kmeans = KMeans(n_clusters=5, random_state=0,max_iter=1000)

kmeansclus_nw = kmeans.fit(X)

predicted_kmeans_nw = kmeans.predict(X)

centers_nw = kmeansclus_nw.cluster_centers_ df['ClusterID_unwt'] = predicted_kmeans_nw

centersdf_nw=pd.DataFrame(centers_nw)

centersdf_nw['ClusterID_unwt'] = centersdf_nw.index

centersdf_nw=centersdf_nw.rename(columns = {0:'Unwt Centroid: Spend in Local',1:'Unwt Centroid: Spend in Organic'})

df_nw=df.merge(centersdf_nw,on='ClusterID_unwt',how='left')

df_nw.head()

Generate a scatter plot for the unweighted clusters:

plt.figure(figsize=(15, 10))

plt.scatter(X[:, 0], X[:, 1], c=predicted_kmeans_nw, s=10, cmap='tab20',marker ='x')

plt.scatter(centers_nw[:, 0], centers_nw[:, 1], c='black', s=500, alpha=0.5); plt.title('Customer Spend - Local vs Organic - Unweighted K-Means',fontsize=18, fontweight='bold')

plt.xlabel('Spend_in_Local',fontsize=15)

plt.ylabel('Spend_in_Organic',fontsize=15)

The clusters look different!

Generate plots to see how the centroid values look like for both cases:

ax = df_nw.plot(x="Total_Spend", y="Pct_Spend_in_Local", kind="scatter",title="Centroid Comparison - Weighted vs Unweighted",

figsize=(10,6),color='plum',s=2)

df_nw.plot(x="Total_Spend", y="Wt Centroid: Pct Spend in Local", kind="scatter", ax=ax, color="blue", s=2,

label="Wt Centroid: Pct Spend in Local")

df_nw.plot(x="Total_Spend", y="Unwt Centroid: Pct Spend in Local", kind="scatter", ax=ax, color="red", s=1,

label="Unwt Centroid: Pct Spend in Local")

ax = df_nw.plot(x="Total_Spend", y="Pct_Spend_in_Organic", kind="scatter",title="Centroid Comparison - Weighted vs Unweighted",

figsize=(10,6),color='plum',s=2)

df_nw.plot(x="Total_Spend", y="Wt Centroid: Pct Spend in Organic", kind="scatter", ax=ax, color="red", s=4,

label="Wt Centroid: Pct Spend in Organic")

df_nw.plot(x="Total_Spend", y="Unwt Centroid: Pct Spend in Organic", kind="scatter", ax=ax, color="orange", s=1,

label="Unwt Centroid: Pct Spend in Organic")

Two out of five clusters have almost same centroid for both cases. For remaining clusters, the centroids for the unweighted K-Means are slightly different than those of the weighted K-Means.

When you are working on an unsupervised learning method, think about the use case carefully and make a decision about whether all the data points should have equal weights or not. Weighted K-Means is an easily implementable technique using python scikit-learn library and this would be a very handy addition to your data science toolbox — the key is to apply the method in a proper use case.