Deep Clustering for Sparse Data. A rather “shallow” and simple approach... | by Alon Agmon | Towards Data Science

We usually cluster data in order to find or learn about relationships and structures that exist within it, especially where our data is too complex or too dimensional for simple descriptive statistics. When it comes to highly dimensional data, it is sometimes striking how informative and useful can be a simple clustering algorithm. But clustering, or unsupervised learning more generally, is often much more challenging than choosing the right algorithm or approach (and that is not simple at all), because a good clustering result usually depends on how the data is organized and represented. Indeed, I have seen more clustering problems solved with a creative way of representing or organizing the same dataset than with some innovative algorithm.
One of the most dominant features of neural networks (if not *the* most dominant one) is their ability to efficiently learn ways to represent highly dimensional data in a manner that reveals exactly those complex relationships and structures that characterize it. This is why neural networks are used so extensively for dimensionality reduction and pattern recognition tasks. Unsurprisingly, over the past few years it has been also suggested and shown that neural networks can assist clustering tasks by providing a better (or more informative) representation of data before it is clustered (See here for some nice overview of DEC and DCN — two of the most dominant approaches).
Many of the proposed architectures that follow this approach are based on some form of preliminary encoding-decoding task (or an autoencoder) which is aimed at creating a more refined representation of the data before it is clustered. Very briefly and simply put, an autoencoder is a network that learns to reconstruct the input that it receives by creating lower-dimensional representations of the data, which (if done right) preserves the most important information about it, and then uses these lower dimensional representations to reconstruct it. The reason that autoencoders are suitable for this task is that if they do a good job in reconstructing their input, then this means that they have learned a lower dimensional representation of the data that probably well captures the structures and relationships within it, and, therefore, using these representations in order to cluster the data will yield a better result.
Unfortunately, many of these proposed architectures are not at all obvious or simple to implement, certainly not for something like a practical POC, and some of which required relatively high resources comparing to their value. However, a couple of months ago I have stumbled upon the paper “N2d:(not too) deep clustering via clustering the local manifold of an autoencoded embedding” . The paper, which didn’t seem to gain much traction, essentially suggests a much simpler and rather easy-to-implement approach to deep clustering. After experimenting with it for a while and getting some very nice results, I thought it might be a good idea to create this brief and practical intro that will allow you to quickly test it and put it to work.
The idea, in short, is based on 3 stages:
(1) Create an autoencoder that will learn lower dimensional representation of the data, which will hopefully capture the most important information and structures within it;
(2) Apply a manifold learning method (such as UMAP or TSNE) to further reduce the dimensions of the data and create finer representations that will improve the performance of the clustering algorithm.
(3) Cluster the data.
I will leave the theory and explanation to the paper (which actually has a lot of practical examples) and delve straight to the slightly modified and shorter implementation, which is mostly based on the code published by authors.
The data I will be using is a rather sparse matrix of users and the number of times they have visited different UI areas of a certain website. My goal is to cluster users into groups, using their activities on the website, in order to better understand their preferences and behavior as a group. The data is normalized on the vector level, has 50 dimensions/features and about 20K users. You can see an excerpt below.
The first thing we’ll do is create the architecture of the autoencoder. Following the architecture presented in the paper, the autoencoder will expand the number of dimensions and then create a bottleneck which will reduce the dimensions to 10 ( a common practice with autoencoders, see here)
The autoencoder can be easily constructed using the following helper function (also given in the paper). The function simply takes the encoding network structure that you want to create as a list (in our case that will be [50, 500, 500, 2000, 10] but this needs to be experimented with and you can certainly use a smaller one) and return a model.
So the following piece of code gives us a full autoencoder that will encode a 50 dim dataset to 10 dimensions.
# X = matrix shown aboveencoded_dimensions = 10shape = [X.shape[-1], 500, 500, 2000, encoded_dimensions]autoencoder = get_autoencoder(shape) 
Recall that the dense layer we are interested in is the “bottleneck” layer which captures the lower (10D) representation of our data (highlighted above). More accurately, what we need here is just the encoder layers which we shall use to create the lower dimensional embedding of our data. So the plan is to train the autoencoder and then just use the encoder layers (with the trained weights) to refine our data. (It sounds complex but if you check the notebook on my repo you’ll see that it's more or less straightforward)
# this will just give us the label we gave the encoder layer# 'encoder_3'encoded_layer = f'encoder_{(len(shape) - 2)}'#we take the layer that we are interested inhidden_encoder_layer = autoencoder.get_layer(name=encoded_layer).output# create just the encoder model that we can use after the autoencoder will be trainedencoder = Model(inputs=autoencoder.input, outputs=hidden_encoder_layer)
So, now all that is left is to fit() the autoencoder and then use the encoder that already references its weights to encode our data to a lower dimensional representation.
autoencoder.compile(loss='mse', optimizer='adam')autoencoder.fit(    X_train,    X_train,    batch_size=batch_size,    epochs=pretrain_epochs,    verbose=1,    validation_data=(X_test, X_test))# use the weights learned by the encoder to encode the data to a representation (embedding)X_encoded = encoder.predict(X)
The second stage is to use manifold learning to refine the clusters and further reduce the dimensions. The reason to add a manifold learning task on top of the encoded embedding, the authors write, is that an autoencoder usually does not take local structures into account, and that “by augmenting the autoencoder with a manifold learning technique which explicitly takes local structure into account, we can increase the quality of the representation learned in terms of clusterability.” Although there are good reasons in favor and against using manifold learning as a prepossessing step before clustering, I believe that the authors are correct (and they show it) that it will probably improve the performance of density-based algorithms and help create denser “neighborhoods.” But this is really something that you need to experiment with on a case by case basis.
In this case, I will follow the authors in the paper and used UMAP for this task using the ( slightly modified) helper function below.
X_reduced = learn_manifold(X_encoded, umap_neighbors=30, umap_dim=int(encoded_dimensions/2))
And now we should have our dateset ready to be clustered.
The next step is pretty much straightforward: just pick your favorite density based clustering method and let it run over the data. Just note that when the data should be visualized in 2d then it is better to lower the dimension of the embedding before visualizing rather than the post-manifold-learning embedding. The data should now be lumped together in a sufficiently dense forms that will make it easier to cluster.
# this is the data that we need to clusterlabels = hdbscan.HDBSCAN(    min_samples=100,    min_cluster_size=1000).fit_predict(X_reduced)# lower dim to 2d so we can plot itreducer = umap.UMAP(n_components=2)embedding = reducer.fit_transform(X_encoded)
As a comparison, this is the result of running HDBSCAN on the data before it was further refined using manifold learning. As you can see, with exactly the same hyper-parameters, HDBSCAN only picked up 2 main clusters, probably due to less refined neighborhoods comparing to the ones produced by UMAP.
It sounds intuitive, but researchers sometimes forget that a good clustering result often depends more on how the data is organized or represented than on the chosen clustering algorithm and its hyper parameters. It is much more challenging to find ways to reorganize a complex data set, but in most cases it pays off.
Deep learning methods usually excel in efficiently learning and producing embedded representations of data, and this is why they are sometimes used as a pre-processing stage for clustering tasks that is aimed at creating a less dimensional and more cluster-able representation of the data. Although deep learning approaches to clustering are often more complex to implement, when the data is highly dimensional then the performance gains they offer cannot be overlooked.
In this post, I have tried to create a brief and very practical intro to a rather simple and straightforward approach (named N2D by its authors). Although N2D is similarly aimed at producing a more clustering friendly encoded embedding of highly dimensional data using DL, I have showed that unlike other approaches, N2D can yield pretty impressive results and performance gains with a much simpler architecture and with relatively little implementation efforts. Although my data set was based on sparse data, which presents a distinct problem in its own right, the authors of the article show pretty impressive results using this approach with many common clustering data sets. To sum, I think that N2D is a very nice addition to the clustering toolbox, especially in the preliminary research stages, which can quickly and efficiently yield some results to work with.
Hope this will be helpful.
The notebook is available in my repo here