KMeans Hyper-parameters Explained with Examples | by Sujeewa Kumaratunga PhD | Towards Data Science

These days anyone with a high school degree can implement a Machine Learning algorithm and that’s a good thing. It is always a good thing when things are accessible to most people. A lot of common libraries are available out there that simplifies the algorithm part. But with great power comes great responsibility. The time freed up writing libraries from scratch should be used to fine tune your model so you get decent results.
Today we’ll look at the tuning of the Scikit KMeans model. KMeans is a widely used algorithm to cluster data: you want to cluster your large number of customers in to similar groups based on their purchase behavior, you would use KMeans. You want to cluster all Canadians based on their demographics and interests, you would use KMeans. You want to cluster plants or wine based on their characteristics, you could use KMeans. All of these problems require unsupervised clustering, that is we do not know what the clusters will look like, a priori. There are other methods for unsupervised clustering, such as DBScan, Hierarchical clustering etc and they each have their merits, but in this post I will address KMeans since it is a computationally light clustering method that you can often run on your laptop, specially with MiniBatchKMeans.
Links to complete code are at the end.
We generate 3 clusters (0, 1, 2) with 2 features (xx and yy). The clusters are generated with the following 3 pairs of x,y values:
mu1 = 3.0sigma1 = 0.5mu2 = 20.0sigma2 = 5.5mu3a = 1.0sigma3a = 0.8mu3b = 5.0sigma3b = 0.4numpoints = 1000np.random.seed(1234)x1 = np.random.normal(mu1, sigma1, numpoints)y1 = np.random.normal(mu1, sigma1, numpoints)x2 = np.random.normal(mu2, sigma2, numpoints)y2 = np.random.normal(mu2, sigma2, numpoints)x3 = np.random.normal(mu3a, sigma3a, numpoints)y3 = np.random.normal(mu3b, sigma3b, numpoints)
The hyper-parameters are from Scikit’s KMeans:
class sklearn.cluster.KMeans(n_clusters=8, init='k-means++', n_init=10, max_iter=300, tol=0.0001, precompute_distances='auto', verbose=0, random_state=None, copy_x=True, n_jobs=None, algorithm='auto')
This is setting a random seed. It is useful if we want to reproduce exact clusters over and over again. We can set it to any number we want. I set it to random_state=1234 below.
We need to provide the algorithm with the number of clusters that we want. Standard literature suggests we use the elbow method to determine how many clusters we need and it works well for Scikits’ cleaned theoretical datasets. In reality, this is only an initial guess. Well, in this example we know we have 3 clusters. So let’s try with n_clusters=3 :
km = KMeans(n_clusters=3, random_state=1234).fit(dftmp.loc[:, dftmp.columns != ‘group’])
We do get 3 clusters, but they are very different from our original clusters. Originally we had 2 clusters in the bottom left, but they are both grouped in to one cluster, the yellow circles. This is because KMeans randomly assigns initial cluster centroids and then tries to group as many points as possible based on the points’ distance to the centroid. Sure, it repeats this process until convergence, but nothing prevents it from being stuck in a local minimum. In fact KMeans is notoriously dependant on the centroid initialization. And here is our first clue that we can not go about blindly using KMeans. If we are asking for 3 clusters, then we need to have some idea where we expect the cluster centers to be in all of the features.
This is where you can set the initial cluster centroids. In our case we have 3 clusters, so we will need 3 centroid arrays. Since we have two features, each of our arrays will be of length 2. So in our case then, we have to have 3 pairs of cluster centers. We know our exact cluster centers since this is a simulation, so let’s try that.
centroids = np.asarray([[mu1,mu1],[mu2,mu2], [mu3a,mu3b]])km = KMeans(n_clusters=3, init=centroids, random_state=1234).fit(dftmp.loc[:, dftmp.columns != 'group'])
Ah! There we go. We have our original 3 clusters for most of the data points!
But wait! That is cheating! We will never know the exact centroids for our clusters. True. We can only speculate what our cluster centers will be approximately. So you don’t have to have the exact centers, but approximate values will help.
What else can we do to improve our clustering specially if we only have approximate cluster centers? Next we will look at another method we can get our original clusters.
I know! You think, but, but, but, we had 3 clusters originally and we set the number of clusters to be 3. What more can we do? How about we set the number of clusters to be twice the number that we expect. What? I know, bear with me, please.
km = KMeans(n_clusters=numclusters, random_state=1234).fit(dftmp.loc[:, dftmp.columns != 'group'])
Look at the bottom right that shows original cluster 0 and 2 — they are more or less well separated. So that is a good thing. But the original cluster 1 (top right) is now separated into 4 clusters. But we know that that is all one cluster from our simulation. So it is a matter of us just consolidating those clusters together after that. Something like cluster (purple, brown, dark green and light green) will become a single cluster.
So if we don’t have an idea of where our cluster centers are, which might be the case with many features, then we can use this trick — of overestimating our number of clusters. One might say this this is overfitting — which is true for the top right cluster, but it is the only way we can get the bottom left clusters to separate, in the absence of good centroids. Even if we had approximate centroids this method will enhance the cluster separation better.
You say: “But wait! I have read you have to normalize your data for KMeans“. Actually that is true. You do have to normalize your data so all the features are within the same range, whenever you do anything that involves Euclidean space, which KMeans does. I was able to get away with it in this case, because my x and y ranges for the original data look about the same : 0–30 (see first graphs in Sample Data section above). But let’s normalize our data to see if it makes a difference.
scl = StandardScaler()dftmparray = (dftmp.loc[:, dftmp.columns != 'group']).valuesdfnorm = scl.fit_transform(dftmparray)dfnorm = pd.DataFrame(dfnorm)km = KMeans(n_clusters=3, random_state=1234).fit(dfnorm)
We see we are still not successful in separating the original cluster 0 and 2 (in the bottom left of the original data), even if we normalize the data.
There are other hyperparameters like tol, max_iter that help with the computational time. These parameters become more important in a more complex problem than what is shown in this example, so I won’t attempt to show them by example.
But let’s look at what they mean:
n_init = By default is 10 and so the algorithm will initialize the centroids 10 times and will pick the most converging value as the best fit. Increase this value to scan the entire feature space. Note if we provide the centroids, then the algorithm will only run once; in fact it will warn us about this at run time. If we set initial centroids or if we set the number of clusters to be more than what we expect (with the intention of consolidating some clusters later on, as discussed above), then we can leave this at the default.
tol = If we set this to a higher value, then it implies that we are willing to tolerate a larger change in inertia, or change of loss, before we declare convergence (sort of like how fast are we converging). So if the change of inertia is less than the value specified by tol, then the algorithm will stop iterating and declare convergence even if it has completed fewer than max_iter rounds. Keep it at a low value to scan the entire feature space.
max_iter = There are n_init runs in general and each run iterates max_iter times, i.e., within a run, points will be assigned to different clusters and the loss calculated for max_iter times. If you keep max_iter at a higher value, then you are guaranteed that you have explored the entire feature space, but often this comes at the cost of diminishing returns.
The other variables decide computing efficiency — so if you have a very large dataset, it would be best that you keep them at the defaults.
It is usually not enough to just run an elbow method, determine the number of clusters and just run the standard KMeans with that. In general we have to explore the data and get subject experts’ opinion on how many clusters there must be and what their approximate centroids should be. Once we have those we can put them all together to tune KMeans:
By providing initial cluster centersBy asking for more clusters than is necessary so we can consolidate some of the clusters in the aftermath.We can also increase the contribution of some features by weighting them more than the others, in an attempt to replace the Euclidean distance by the Mahlanobis distance. For example in the above example if we thought xx is 10 times more important in the separation than yy, then we would multiply xx by 10 after the normalization step. Though this is not always advisable — there are other ways to deal with this like the PCA for example.
By providing initial cluster centers
By asking for more clusters than is necessary so we can consolidate some of the clusters in the aftermath.
We can also increase the contribution of some features by weighting them more than the others, in an attempt to replace the Euclidean distance by the Mahlanobis distance. For example in the above example if we thought xx is 10 times more important in the separation than yy, then we would multiply xx by 10 after the normalization step. Though this is not always advisable — there are other ways to deal with this like the PCA for example.
The complete python code to reproduce the above plots can be found here. The pdf version of the Jupyter notebook with the plots can be found here.