Interpretable K-Means: Clusters Feature Importances
By: Yousef Alghofaili
Tags: K Means, Model Interpretability, Editors Pick, Keyword Extraction, Feature Importance

Machine learning models go through many stages for them to be considered production-ready. One critical stage is that moment of truth where the model is given a scientific green light; Model Evaluation. Many evaluation metrics are designated for different purposes and problem specifications, but none of them is flawless. Hence, data scientists have a huge burden to carry when choosing which evaluation metric to use that best fits the problem at hand and serves as a functional decision-inducing value for a machine learning project.

However, some evaluation metric values could seem unintuitive and cannot be explained in laymen’s terms, especially when evaluating unsupervised clustering algorithms using internal indexes (No external information is present such as true labels). Although these measurements serve a considerable benefit for comparative benchmarking, combining them with an interpretation is crucial for independent, intuitive validation and easier results communication with stakeholders.

Intuition vs. Internal Validation Indexes

Data scientists tend to lose a focal point in the evaluation process when it comes to internal validation indexes, which is the intuitive “Human” understanding of the model’s performance and its explanation. To elaborate by a counterexample, supervised classification evaluation metrics such as Accuracy, Precision, or Recall have very intuitive explanations for both laymen and experts. For instance:

Accuracy : “We have correctly classified this percentage of samples out of all samples we have.”

: “We have correctly classified this percentage of samples out of all samples we have.” Precision : “We have correctly classified this percentage of Positives out of all predicted positives.”

: “We have correctly classified this percentage of Positives out of all predicted positives.” Recall: “We have correctly classified this percentage of Positives out of all actual positives.”

On the other hand, internal validation indexes such as the widely used Silhouette Coefficient, Calinski-Harabasz Index, Davies-Bouldin Index, and many others are, more often than not, used comparatively rather than an inherent and independent performance evaluation. The values of these measurements can be broken down to how compact each cluster is (How similar the cluster data points are to each other), how well-separated the clusters are (How dissimilar each cluster’s data points are from other clusters’ data points), or both. But, these measurements are not as easy to explain and connect to the clustering problem you are solving.

Notwithstanding how valuable these measures are, obtaining an interpretation for a model along with an internal evaluation index is crucial when trying to understand how good your model is as a data scientist or attempting to deeply communicate your results to stakeholders where most internal validation indexes do not pass the ELI5 (Explain it Like I’m 5) Test :). This article will focus on one of the most popular unsupervised clustering algorithms; The K-Means, and presents two possible techniques to extract the most important features for each cluster. The article’s outline is as follows:

If you already know how K-Means works, jump to the Interpretation Techniques section, or would like to visit the repository for this article and use the code directly, visit kmeans-feature-importance.

K-Means Business Value

Say that you are running a business with thousands of customers, and you would want to know more about your customers, albeit how many you have. You cannot study each customer and cater a marketing campaign specifically for them, for example. Yet, you know that each customer will probably have different demographic (e.g., Age), Geographic (e.g., Region), Psychographic (e.g., Spending Habits), and Behavioral (e.g., Engagement) properties. Segmenting them into multiple similar groups will simplify understanding who they are from the most prevalent properties of each group. The most popular algorithm to solve this problem, I am sure you guessed it! Is K-Means; “K” will refer to the number of possible customer segments/groups/clusters, and “Means” can be thought of as imaginary persons who are the center (Most similar to their group members) of each group in the K groups.

How K-Means Works

K-Means is an unsupervised clustering algorithm that groups similar data samples in one group away from dissimilar data samples. Precisely, it aims to minimize the Within-Cluster Sum of Squares (WCSS) and consequently maximize the Between-Cluster Sum of Squares (BCSS). K-Means algorithm has different implementations and conceptual variations, but for the sake of this article, We will focus on the most common method, namely Lloyd’s algorithm (Naive K-Means), which follows an iterative approach to find a sub-optimal solution. Before starting, we will prepare a simple dataset for explanation and see how it looks like.