A Fresh Look at Clustering Algorithms
By: Dmitry Selemir
Tags: Clustering, Machine Learning, Python, Video Tutorial, Data Science

Photo by Guillermo Ferla on Unsplash

A Fresh Look at Clustering Algorithms

A deep dive into a new methodology for cluster recognition.

The project is by no means in its final stages. Sharing the current thinking and going through code in some detail to get some feedback and see if this is worth developing further. Born out of frustration with the current methods ( at least the ones available via standard libraries), particularly the adhoc nature of all the main approaches, including the crowd favourite KMeans. I am not claiming that the proposed code is done in the most efficient manner possible, but I certainly made an effort to get there. Tips on improvement would be greatly appreciated.

Video tutorial

Setting the problem

Let me start by providing a little bit of context. Since Medium is a blog platform rather than a place for scientific research, I can afford to tell the story rather than present the results in a very dry form. Please forgive me if that disappoints you — this is essentially a bit of a ‘show and tell’.

A couple of years ago I was trying to solve the problem of analysing the raster files. In case you have never worked with these before, take a look at my Geospatial adventures series for more detail:

In a nutshell — these are matrices of height numbers acquired by flying over the area you are trying to analyse, firing a laser beam at it at regular intervals and measuring the response. The resulting matrix can be used to extract the information about various objects on the surface. These objects can be anything from mountains and trees to buildings and bridges. I was specifically interested in extracting building information, which meant that I had collections of closely packed points, which I somehow needed to interpret as polygons. If you have a lone, free standing building in the middle of a field — it’s easy, all you have to do is apply convex hull algorithm out of the box (from shapely library. Again, if you are new to this — take a look at my primer here). But what if we are looking at a densely populated city centre area?

I ended up solving this by writing a custom clustering algorithm, which essentially treated collections of dots representing the buildings as a graph, where each node (our raster pixel) is connected to it’s immediate neighbours if they meet the height criteria. The problem of recognising the building polygons was essentially solved by counting unconnected graphs and then applying the convex hull to them.

Now, with the COVID crisis upon us and with plenty of spare time on my hands while job hunting, I thought it would be a good idea to revisit the problem and generalise the algorithm. What I am presenting here is the result.

Phew… That was kind of a long introduction. Let’s dive in.

One of the fundamental problems with clustering is, we are really good at judging it visually and we rarely think about the mechanics of it. As a result, it is really easy to find ways of confusing the casual observer into jumping to a false conclusion. If I zoom in on a set of points on a 2D graph — I might say that they are fairly distributed, if I zoom out — they appear as a cluster. So from the very beginning, we need to define the scaling of the problem, this will be one of the main parameters driving the model.

The easiest way to do that is by defining the size of an elementary cell in the space of our problem. If this is a simple 2D, this means that we spread out all of the points in our dataset in the bounding square containing this dataset and draw square cells around each point. If we are talking about a more complex, n-dimensional problem, for example a collection of word2vec text representations — this will be a unit volume cell in n-dimensional space. It’s a lot harder to spot groupings of data when you can not see their graphical representation, so having the right way of defining the scale is really important.

If you have categorical variables, they can always be reduced to a set of binary variables (1/0) or you might opt for a more complex version using real numbers if you want to build in some meaningful relationships between them. In any case, an approximation for the number of required cells/divisions for each dimension can be constructed as:

Where N is number of points (vectors) in the dataset, n is number of dimensions (continuous variables) and k is number of categorical variables. I am assuming here that categorical variables are reduced to just 1/0.

The intuition behind this number is: going significantly above it would result in divisions being too small and therefore unless the clustering is very pronounced, you wouldn’t be able to detect it. Going significantly below it risks making the cells so large that you just gloss over any irregularities in the data. In my buildings problem example this is particularly important as, in order to detect all the buildings, we want to have the number of divisions quite a bit below the critical one. This is because the distance between our points is the same everywhere, what changes is their type (their height profile). Ultimately, we make the divisions large enough to include two layers of neighbouring points.

For the purposes of this post, I am not going to deal with multi-dimensional problems, I might come back to it some other time. The code presented here should be able to deal with them though, with small modifications if you want to include the categorical variables. After all, the purpose here is to demonstrate the principle.

The algorithm

To give you a very condensed, nutshell version of what we are doing:

Normalise the data, so each variable is now in a 0 to 1 interval. Multiply each variable by D and take the integer part. This effectively standardises each point, by placing it into a specific grid cell. Count number of points in each grid cell and discard the cells with numbers below specific threshold. This is another important parameter we are going to pass into the model. In this implementation, I am looking at specific percentiles, but it can be driven by an arbitrary number. The resulting structure can be represented as a collection of trees(or rather graphs, but lets roll with trees) and all we have to do to identify our clusters is run a breadth first search (BFS) algorithm to count the instances of unconnected trees. Finally , once the main clusters have been identified, the points which were left out, as their cells didn’t pass the threshold, need to be attributed. Here, I am doing this outside of the main function by calculating the minimum distance to centroids of identified clusters. This is not the only option and I believe not even the best option. It will do for now though and (!) it’s easy and quick to implement.

OK, let’s get down to it.

Let’s start with creating a dummy dataset we are going to be playing with. Towards the end of this post I will also share the comparison with the results of running this dataset through some of the standard clustering algorithms from scikit.

Loading the libraries we would need:

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

pd.plotting.backend='seaborn'

import time

import itertools

Most of the time I am going to try and avoid using pandas, it will be used in the very beginning as we set up and then in the end to help with visualising and to keep everything in one place. Throughout the bulk of it, we are going to work with numpy arrays, lists and dictionaries. Keeping it simple (and helping with performance a bit). As a side note — I have played with acceleration of the code using numba and cython, but I wasn’t impressed with the results, so for the time being we are going to leave them aside.

Matplotlib and Seaborn for visualisation, also using Seaborn as a backend for pandas. Using time module to look at comparative performance and using itertools to avoid loops (well, at least in the explicit python form).

The dataset:

Essentially these are four clusters, with points nicely bunched together toward the centre of each cluster + some basic stretching on top of it so they don’t appear too regular. They also end up overlapping a bit so will be good to see how we deal with those cases.

We are going to normalise them:

And now go ahead and visualise:

plt.figure(figsize=(10, 10))

sns.scatterplot(ttf.x, ttf.y, alpha=0.5)

Sweet.

In our case, the number of cells can be calculated as:

D = int(np.sqrt(len(ttf_norm)))

tolerance = 50

This gives us D=63. So model parameters we can use would be D = 60 (just going for a nice round number) and we are going to look for 50th percentile in terms of number of points per cell (tolerance). In general — the choice for tolerance depends on how well our data is distributed. If there is no way of checking it visually or otherwise — It’s not a bad idea to run the model at different tolerance levels and see how many clusters are found for different levels and which result makes the most sense (is most stable).

We’ll also convert our input dataset into a numpy array. In this case it will have just two columns, in case of more dimensions/variables it will have more.

X = ttdf_norm[['x', 'y']].values

OK, we are ready to build. I am going to go through it step by step.

Scale and discretise:

Xint = (X * D).astype(int)

2. Create string ids for each cell:

Xstr = np.apply_along_axis(

''.join,

1,

np.char.zfill(

Xint.astype(str),

len(str(D))

)

)

What’s happening here is, we are converting each coordinate into a string and then pad it with preceding zeros so that each one is the same length as the string representing the number of divisions (D). Final step, merge them into a single ID. The reason I am creating IDs this way is, this gives me an ability to home in on the nearest neighbours of each cell without having to do expensive lookups in large arrays, I can just infer their ids from the current id.

3. Create the dictionary with frequency of occurrence counts for all cells. As easy as :

unique, counts = np.unique(Xstr, return_counts=True)

tree = dict(zip(unique, counts))

4. Calculate the threshold:

mincount = int(

np.percentile(

np.array(

list(

tree.values()

)

)[

np.array(

list(tree.values())

) >1

],

tolerance

)

)

This one is a bit busy, so let’s see what is going on. We are creating a numpy array out of all the values in our tree and discarding everything with one point or less. We don’t want these to influence the percentiles too much as they don’t bring any value. Out of the resulting array, we calculate the value corresponding to the percentile set by our tolerance variable. The simplest case of 50% corresponds to median.

5. Pruning the tree: get rid of everything below the mincount. Using simple dict comprehension:

tree = {k: v for k, v in tree.items() if v > mincount}

As a side note — when playing with numba.jit I had to replace this with a straightforward python for loop (my heart bled as I did so) — as numba can not handle list/dict comprehension in the current implementation. Now, thinking about this — this can be done without loops at all using numpy arrays for key and values. This should make the code faster, although this isn’t the main bottleneck, so this won’t change the calculation time materially.

The main part is already done. What we are left with is a dictionary with keys corresponding to all meaningful cells, which need to be counted.

Essentially, we need to perform a breadth first search on these. What this means is we are going to select a random id from our tree, mark it as visited and assign all its neighbours to ‘tocheck’ list (queue in classic implementation), we then go select any value from ‘tocheck’ list and repeat the same process. In the classic implementation this needs to be the first value from ‘tocheck’ hence the use of queues, but, in our case, we are going to have to go through all of them, so the order is immaterial. Once we run out of items in ‘tocheck’ list we have completed our first tree. We can assign it to the first cluster and pick up the next id from the original list.

I am not going to go through the details of exact implementation here as this is not particularly novel or interesting — I am still trying to make it faster as this is where most of the time is being spent. I will, however, show you the neighbour selection, since this part is not trivial (especially without loops).

First, we generate the list of shifts for our coordinates. We are looking for all nearest neighbours one space away. In 2D case — that’s 9 points (including the origin, in 3D — it’s 27 and so on. We need to be able to perform the shifts in any number of dimensions defined by X. This is where itertools comes in handy.

l = list(itertools.product([-1, 0, 1], repeat=Xint.shape[1]))

l=list(map(list, l))

l.remove([0] * Xint.shape[1])

What’s happening here is — first we create all possible permutations of -1, 0 and 1s in the horizontal dimension of X matrix (in our case 2, as we only have 2 variables).

These are returned as tuples, so we need to convert them to lists — that’s the job of the second line.

Final step — we remove the zero shifts as it’s not useful for us.

Here’s the result:

[[-1, -1], [-1, 0], [-1, 1], [0, -1], [0, 1], [1, -1], [1, 0], [1, 1]]

Now, let lD be the length of string representation of our D parameter and ids be the single id we are trying to find neighbours for. I know, using id would be more logical, but Python has dibs on it, so we have to do with ids instead.

nbrs = np.array(

[

list(

[

int(

''.join(

list(ids)[i * lD: (i + 1) * lD]

)

) for i in range(len(l[0]))

]

)

] * len(l)

) + np.array(l)

OK, lots going on here, so to break it down: Taking a list of a string — converts it into a list of characters making it up. We then take portions of that list with number of elements corresponding to the length of our D parameter (lD) and glue them together into strings using join. Once that’s done we convert them to integer values again and that gives us a replica of the corresponding line from X matrix, i.e. all the variables corresponding to this unit cell. We then replicate this list as many times as the number of neighbours we are trying to get to (number of stresses in the l list). Final step — convert both the resulting list and the list of stresses to numpy array and add them up. This gives us element-wise sum and hence the coordinates of all neighbours. Note that this would work for any number of variables, so this is good for n-dimensional vectors. The problems will potentially start when get into really large numbers of variables, but let’s not worry about that for now.

Finally, we repeat the conversion to ids:

nbrs = np.apply_along_axis(

''.join,

1,

np.char.zfill(

nbrs.astype(str),

lD

)

)

And voila, we have the ids of neighbours. All that’s left is a quick check that we haven’t visited them before and that they do indeed belong to the identified tree.

Look ma, no ‘for loops’!!!

Erm.. Actually there was one list comprehension there, even though it is quite a short one. Still, if you figure out a way to do it without it — please let me know!

Once we’ve done our tree counting , we are more or less done. All we have to do is attribute the unit cells we left out and then mark up the corresponding points.

Let’s look at some pictures!

Time to compare the results of running our function with the standard clustering methods. If you want to review the latter, here’s a very good post with a whole bunch — 10!

In fact, we are going to run all of them on our dataset (apart from the optical one, since it’s rubbish) and then compare them to running the method described here (I call it DenseTrees, since we are looking for high density areas using tree structure).

Just to give you a quick example of running it for one of the methods

Load the model:

import sklearn

from sklearn.cluster import AffinityPropagation

model = AffinityPropagation(damping=0.9)

And run the fit:

%%time

model.fit(X)

The first line is to get the execution time.

Final step

ttf_norm['Affinity'] = model.predict(X)

On my laptop this particular method took the longest — around 21 seconds.

Same process with the others. I have played with the parameters driving them a little bit to get the best result out of the box. In case of Affinity — it didn’t really help much.

I have to stress that some of the methods — KMeans (straight and minibatch), Spectral, Gaussian mixture, Agglomerative and Birch, all have number of clusters as an input, which, in a way, defeats the purpose. You have to know the number of clusters before applying them. In reality, we can discard them straightaway as they won’t be useful for any interesting problem we are going to try and solve. Let’s keep them in, just for the sake of the exercise though.

Plotting away:

You can see that MeanShift, Birch, DBSCAN, Affinity all performed poorly. Incidentally, DBSCAN was particularly fiddly in terms of getting the right parameters to even get to this stage. Gaussian mixture looks like the best of the whole lot. Our DenseTrees method performed on the par with KMeans. If we change our second path attribution we can probably get as good a result as from GM (in fact we can use GM for the final attribution). Incidentally Gaussian Mixture was about 6x faster than DenseTrees, so there’s some room for improvement there too (we still did OK with 300ms — 6th fastest). The main outtake here — we have outperformed every algorithm which didn’t have the number of clusters as an input and we are not far off the best algorithm from the ones that did. Not too shabby.

Final note

I did mention that the problem of identifying the buildings from raster files served as the inspiration for this, so it would be a shame not to show how the algorithm performed against that dataset.

Just to give you some context. After some filtering and cleaning the raster file plotted as a scatter plot looks like this:

Extra kudos to those who can figure out where this is.

These are just points plotted on a normalised scale. Anything below 10 meters is set to zero. You can just about see that we caught some trees in our net. Not much we can do about that unfortunately. At least for now. This square has 2000 by 2000 resolution (so 4mln datapoints in total) and each pixel corresponds to 0.5m by 0.5m square. We have to go pretty granular to catch the gaps between buildings, so to process this I used D=500 and tolerance of 10%. Note that critical value for this would be too much as all points are evenly spaced already. On my laptop this took about 5 hours to process. Not great. The original optimised for 2D algorithm took about half an hour, so there’s definitely plenty of room for improvement.

The algorithm returns 247 clusters. And here they all are in all their rainbow- coloured glory:

You can see that we mostly got lucky with the trees anyway — all the grey dots are the ones left out. For this problem there is no need for the extra attribution step, which is why I am leaving it out of the main function.

That is all for today. Looking forward to your criticisms and suggestions.