How To Evaluate Unsupervised Learning Models | by Callum Ballard | Towards Data Science

So far in this blog series, we have looked at how to create automated playlists of songs by clustering a collection of tracks, based purely on their audio features. Previously, we worked on a toy example of 32 songs and showed how Hierarchical Agglomerative Clustering (HAC) can automatically create sub-groups of similar songs. We were able to verify the results of this clustering exercise through our existing knowledge of the songs (our algorithm confirmed that Motörhead and Black Sabbath are musically similar — go figure).
But what if we don’t have such prior knowledge of our data? What if the data isn’t even labelled (as is the case in many real-life clustering cases)? Even if it is, what if these labels are initially meaningless to us? There are plenty of artists that I’ve never even heard of, and if we’re trying to group thousands of tracks then it’s clearly impractical to manually verify every cluster. In these cases, we need some kind of mathematical measure for how ‘successful’ our clustering has been.
To explore how we might go about doing this, we once again turn to Spotify’s API. Suppose we take the songs from four very different playlists:
Rap UK
Smooth Jazz
Classical Essentials
Essential K-Pop
If we merge these into a single dataset, an unsupervised machine learning algorithm ought to be able to group their songs into four clusters that somewhat resemble the original four playlists.
Before we get started, we may indeed want to check our assumption that songs from these diverse playlists are indeed ‘different’. Of course, visualising more than two features simultaneously on a single chart is challenging. However, if we plot a scatter matrix of all features, colour-coded by playlist, we can see that there are plenty of metric combinations that demonstrate the distinct characteristics of each genre.
We saw in the previous blog that running a clustering algorithm in Scikit-Learn is very simple:
#We scale the data to ensure that#feature units don't impact distancesfrom sklearn.preprocessing import StandardScalersclaer = StandardScaler()X_scaled = scaler.fit_transform(X)#This scaled data can then be fed into the HAC algorithmfrom sklearn.cluster import AgglomerativeClustering#We can tell it how many clusters we're aiming foragg_clust = AgglomerativeClustering(n_clusters=4)assigned_clusters = agg_clust.fit_predict(X_scaled)
We also saw that HAC has three different ‘linkage criteria’ — the method by which the algorithm iteravely joins clusters together:
ward (default): picks the two clusters to merge in a way that the variance within all clusters increases the least. Generally, this leads to clusters that are fairly equally sized.
complete (or maximum linkage): merges the two clusters that have the smallest maximum distance between their points.
average: merges the two clusters that have the smallest average distance between all the points.
Let’s see how these three linkage criteria deal with the genre playlist dataset. We represent our results in a matrix, showing what percentage of each playlist’s songs have ended up in each cluster (the clusters have been named, arbitrarily, ‘A’, ‘B’, ‘C’, and ‘D’).
Of course, if the clustering had been pefect, we’d expect each row and each column of the matrix to contain exactly one entry of 100% (it needn’t be in a diagonal, of course, since the cluster name assignment is arbitrary).
It’s worth mentioning that there is another common type of clustering, K-Means, which works slightly differently. Whereas HAC iteratively decreases the number of clusters by merging them together, K-Means clustering maintains a fixed number of clusters (the titular k), but iteratively changes the members of each cluster.
K-Means is simple to implement in Python:
#We scale the data to ensure that#feature units don't impact distancesfrom sklearn.preprocessing import StandardScalersclaer = StandardScaler()X_scaled = scaler.fit_transform(X)#This scaled data can then be fed into the K-Means alorithmfrom sklearn.cluster import KMeans#We can tell it how many clusters we're aiming forkm_clust = KMeans(n_clusters=4)assigned_clusters = km_clust.fit_predict(X_scaled)
If we apply K-Means clustering to the playlist dataset, we get the following:
While these matrices are good for ‘eyeballing’ our results, they’re far from mathematically rigorous. Let’s consider some metrics to actually help us assign a number to our cluster quality.
The Adjusted Rand Index is a variation on the classic Rand Index, and attempts to express what proportion of the cluster assignments are ‘correct’. It computes a similarity measure between two different clusterings by considering all pairs of samples, and counting pairs that are assigned in the same or different clusters predicted, against the true clusterings, adjusting for random chance.
This (as well as the other metrics we’ll consider) can be evaluated using Scikit-Learn.
from sklearn import metricsmetrics.adjusted_rand_score(predicted_labels, actual)
The Adjusted Rand index is bounded between -1 and 1. Closer to 1 is good, while closer to -1 is bad.
We see that K-Means and Ward Linkage have a high score. We’d expect this, based on the matrices we previously observed.
The Fowlkes Mallow Score is similar, in as much that it tells you the degree to which cluster assignments are ‘correct’. In particular, it calculates the geometric mean between precision and recall. It’s bounded between 0 and 1, with higher values being better.
metrics.fowlkes_mallows_score(predicted_labels, actual)
We similar rankings to the Adjusted Rand Index — which we would expect, given that they’re two methods of trying to answer the same question.
It’s worth noting that we needed to know the original labels in order to compute these metrics. Given that dealing with unlabelled data is one of the main use cases of unsupervised learning, we require some other metrics that evaluate clustering results without needing to refer to ‘true’ labels.
Suppose we have the following results from three separate clustering analyses.
Evidently, the ‘tighter’ we can make our clusters, the better. Is there some way to ascribe a number to this idea of ‘tightness’?
The Silhouette Score attempts to describe how similar a datapoint is to other datapoints in its cluster, relative to datapoints not in its cluster (this is aggregated over all datapoints to get the score for an overall clustering). In other words, it thinks about how ‘distinct’ the clusters are in space — indeed one could use any measure of ‘distance’ to calculate the score.
It is bounded between -1 and 1. Closer to -1 suggests incorrect clustering, while closer to +1 shows that each cluster is very dense.
metrics.silhouette_score(scaled_feature_data, cluster_labels)
We see that none of the clusters have super-high Silhouette Scores. Interestingly, we see that the Average Linkage clusters have the highest scores. Remember, however, that this algorithm produced two clusters that each contained just a single datapoint, which is unlikely to be a desirable outcome in a real-world situation (a lesson that you often can’t rely on a single metric to make decisions about the quality of an algorithm!)
The Calinski Harabaz Index is the ratio of the variance of a datapoint compared to points in other clusters, against the variance compared to points within its cluster. Since we want this first part to be high, and the second part to be low, a high CH index is desirable. Unlike other metrics we have seen, this score is not bounded.
metrics.calinski_harabasz_score(scaled_feature_data, cluster_labels)
Here we see that our K-Means and Ward Linkage algorithms score highly. The Complete and Average linkage algorithms are punished for having one or two large clusters, which will have a higher level of intra-variance.
This is the latest blog in my ‘Music By Numbers’ column, which uses data to tell stories about music. I’d love to hear any comments about the above analysis — feel free to leave a message below, or reach out to me through LinkedIn!