Use Gaussian Mixture Models to Transform User-Item Embedding and Generate Better User Clusters
By: Alon Agmon
Tags: Clustering, Machine Learning, Deep Learning, Reccomendations, Unsupervised Learning

Use Gaussian Mixture Models to Transform User-Item Embedding and Generate Better User Clusters

Improve clustering of user-item embedding by using Gaussian mixture model to generate new and tighter user features Alon Agmon Apr 16·14 min read

Photo by Markus Spiske on Unsplash

I. Intro

A large company that sells music records hired you to help them better understand their customers and their preferences so that they can personalize their service. The company is mainly interested in understanding the kind of groups that exists in their audience, their listening patterns, and their preferences. For this task they have given you their most precious artifact — their complete user-artist interaction data set.

User-item interaction data is a common way of representing different forms of engagement between users and items, such as movies users watched, ads users clicked, or shopping items users purchased. It is considered a somewhat raw but useful form of input for tasks involving finding similarities between users on the basis of the items they engaged with, or predict the items they will engage with in the future.

Raw interaction data consisting of 9 users and 4 items. Another common way of representing the same data is with a list of user-item pairs.

Because raw interaction data is usually quite sparse and noisy, it is a common practice to first embed it in a lower dimensional space in order to maintain only the important or relevant information about each item or user. The lower dimensional representation of the data can then be used for a variety of downstream tasks, such as classification or clustering of users on the basis of similarities among the items they choose.

There is more than one way to embed user-item interaction data in lower dimensional space. The simplest is to run it through a dimensionality reduction algorithm, such as PCA, while maintaining as much variance as possible. It might be argued that the output is not really an embedding, though it can be similarly used to cluster the data and find similarities between users.

Another very common (and often much more accurate) strategy is to use some form of matrix factorization in order to decompose the user-item matrix (N_users * N_items) to 2 lower dimensional matrices — N_users * N_latent_factors and N_items * N_latent_factors — the product of which is the original user-item matrix. The resulting latent dimensions will capture the most relevant information or characteristics about each item and therefore improve the performance of the downstream clustering task.

Lower dimensional embedding data of 3 Users and 3 latent factors

Lower dimensional embedding data of 3 Items and 3 latent factors

Suppose that indeed you were able to create a lower dimensional embedding layer that uses just 50 dimensions to accurately represent about 4,000 artists (or items) that exist in the dataset you were given. However, most of the clustering algorithms that you try still fail to find any meaningful clusters in the data, which seems too dense to be clusterable in a meaningful way.

This is a pretty common scenario. As usage is intense and as the number of items is higher, so the dimensionality and complexity of the embedding layer usually also increases. When you have a few thousands of heavily used items, then its quite likely that your embedding layer will consist of at least a few dozens condensed dimensions, which will not always cluster very well (although they might be extremely useful for other kinds of tasks). You can obviously embed your data in any number of dimensions, but at a certain point you might start losing important information about the distinctions between users if you get too low.

In this post I would like to suggest a method to further refine hard-to-cluster user embeddings in order to improve the performance and outcome of their clustering. In short, it is based on learning Gaussian Mixture model from item embedding data and then use it to generate new user vectors based on the probability of each user to be part of each item cluster distribution. There is nothing especially ingenious in the proposed approach besides the fact that it is simple and proved useful in a sufficient number of experiments to make sharing it a good idea. Although it is aimed at the use-case of clustering user-item embedding, because it relies on the availability of both item embedding and user embedding that share the same vector space, it can certainly be extended to other contexts.

The post proceeds in 4 main sections. Section II shows one method of generating embedding from user-items interaction data (although there are many more). Section III tries to cluster the data and showcase the main problem in clustering such embedding. Section IV presents and implement our proposed approach. Section V concludes.

II. Embedding Interaction Data

The data set I will use here is LastFM 360K. As you can see below, it can be easily formatted as a plain user-item interaction data.

Unfortunately, I cannot use any form or subset of the original datasets we used to test the approach presented here, but the LFM’s data is both publicly available and seems to be a good alternative to make the point. It is important to note, however, that the LFM dataset is different in a few respects that actually make the results less impressive than the original experiments we run. To give just one noticeable example — LFM’s data set actually registers users, items and number of uses. This means that every “row” or user-item pair should be weighted by the number of interactions reported to get accurate results. I have chosen to ignore these complications for the sake of simplicity but this should be obviously taken into account in real scenarios.

So the first stage is to generate the embedding. The embedding model that I have actually used is Word2Vec. Although W2V is a model that was originally designed to create embedding of words on the basis of their appearance in sentences, it is also known to do a good job in creating embedding of many kinds of sequences, including embedding of items on the basis of their appearance in user’s “shopping baskets”. I will not get into too much details about how to generate the embedding as the method described here can be applied to any embedding regardless of how it was generated, and, also, I believe that the relevant code in the attached notebook is pretty self explanatory and reproducible.

In short, however, I have used (Gensim’s) W2V in order to learn embedding of artists on the basis of their appearance in users’ history. In other words, we will feed the model with a list of users and a list of artists each user has listened to, and the model will generate for each artist a vector of latent factors that represent its most important characteristics.

For example, since one of the artists in the list is ‘robert schumann’, our trained model would generate a corresponding embedding record that represents ‘robert schumann’ with a vector in a space consisting of all the artists in our dataset and, if it is accurate, “position” it closer to vectors of similar artists — i.e., artists that usually appear together with Schumann among users with a similar history.

The embedding vector of Schuman

Although there are more accurate methods to verify the accuracy of embeddings, it is also a good idea to verify it using a few examples that can be visually tested. As shown below, we would indeed expect an embedding learnt from a dataset of music artists and listening users to detect similarities between artists of the same genre.

So, now that we have our accurate embedding matrix we can continue with the task of clustering our users and understanding better the kind and structure of the groups that exist in our data. However, one of the disadvantages of using W2V for embedding of interaction data is that it only leaves us with embedding of items or artists while we are interested in clustering users. One way to deal with this is to take all the vectors of the items or artists that each user has engaged with and take their mean or median. In other words, if i have listened to Schumann and Miles Davis then my user will be represented by a vector equals to:

(model.wv['robert schumann'] + model.wv['miles davis']) / 2

It will take a long post to detail the advantages and disadvantages in this method or to review other methods, but it is a rather efficient way of capturing users’ “position” in the relevant latent space. The resulting vector above will be somewhere “between” Schumann and Miles in a manner that describes my preferences and the preferences of users like me. In the code below, for example, I generated a mean vector for a user that listened to 2 classical artists (Schumann and Mahler) and 2 jazz artists (Miles Davis and Sonny Rollins). You can see that the 10 most similar artists to this user’s mean indeed consist of a mix of jazz and classical artists.

Accordingly, we will create the users’ matrix by taking the mean of all the artists that each user has listened to.

III. Clustering Users

So now we are set to try and cluster our users. We will use HDBSCAN, which is an efficient density-based clustering algorithm because we want the algorithm to determine the number of the clusters in our data according to the way it is structured rather than making assumptions about the shape of the data and its distribution, and also because it is a relatively fast and efficient. Unfortunately we are not very successful.

Unsurprisingly, HDBSCAN has determined that, besides a few small clusters, most of our data is noise, and the 2D projection above (of the user embedding) somewhat explains it. Our data is pretty dense, meaning that users are not that easy to distinguish a part. This result is quite common and usually requires more experiments with different transformations of the data and hyper-params tuning.

At this point, some will argue that the attempt to “tear up” clusters is somewhat artificial or futile because the reason that the data is lumped together in this way is… because this is how the relationships and distances between data points look like. However, it is crucially important to remember the problem at hand. Our task is to segment users that share similar preferences and behavior patterns. In other words, it is the purpose of the task to tear the data apart to clusters. However, because our data is so dense then we need to find a way to bring similar users even closer than they really are, and push different users even further away from each other.

As mentioned above, the purpose of this post is to add one more tool to your arsenal of ways to deal with such situations. Beforehand, however, I just want to briefly showcase 2 other approaches or transformations that will serve as a baseline.

First, although it is somewhat controversial, one very useful approach to refine “big lumps” of data is to use a manifold learning algorithm, such as TSNE or UMAP, in order to better represent local structures and further refine the clusters by bringing similar samples closer . I have chosen to use UMAP due to its good performance with large Ns and the balance it exhibits between local and global structures.

As you can see below, there is some improvement in the way our data is now represented as UMAP indeed further refined the clusters by tearing them a part abit, allowing HDBSCAN to find denser strctures. But, still, most of our data is lumped together.

Another common (and simpler) method to break up clusters is simply to take the log and normalize the data. I'm ignoring a lot of theory here, but the main part is that applying this kind of transformation using the code below will leave us with the following structure.

using log transformation and normalizing to separate clusters

This method also seems like an improvement, though it is much better separated in 2D space then in multi-D space. In other words, this transformation would be helpful if the clustering is to be performed in 2D but that means that a lot of data is going to be missing, which means that the value of the result will depend on the problem at hand and the data.

III. From Clusters to Distributions

Let me briefly explain the logic of the proposed transformation and then delve right into the implementation.

We have a vector space that contains both item vectors and user vectors (where each user vector equals the mean of the item vectors the user engaged with). We want to bring similar users closer together. Since what makes users similar is the fact that they listen to the same items or artists then, in other words, we want users to be closer to the items they engaged with and farther from the items they don’t. This has led us to try the following simple approach presented here.

Cluster item vectors by learning a mixture model based on the assumption that each item was sampled from a Gaussian distribution. When this stage is complete, we have a K number of item clusters, while each cluster represents a distribution of items. A good result will generate clusters of similar items, such as clusters of rock bands, pop artists, etc. But even if this clustering is not accurate it is only used to transform the data (or essentially generate new features from it). Lets call the results of this stage the item clusters. Create a vector for each user based on the probabilities of its being part of each item cluster (or part of its distribution ). For example, suppose that in step 1, we found two item clusters — one of classical music artists and another of jazz artists. In step 2, we iterate over our user vectors and estimate the probability of each user vector being part of each item cluster. In other words, for each user we estimate the probability that its vector was sampled from the classical artists item cluster and the probability that its vector was sampled from the jazz artists items cluster. To simplify things:

Assuming we have 2 clusters (or distributions): Z1 and Z2, and user vector X1 The new vector of user 1 will be [P(Z1|X1), P(Z2|X1)]

Why is this helpful? I cannot provide a mathematically rigorous proof (yet), partly because it is not always true, but the following example might be a starting point to explain the intuition.

Consider vectors A, B and C in the image below (each is represented by a data point in 2D space) and assume that the distance between B and C, as well as between A and B, is equal (or nearly equal). Its obviously hard to determine their relative similarity (or dissimilarity) comparing to other vectors around them based on their position in space. In other words, the question “is B more like C or more like A?” is hard to answer due to their nearly equal distance. That’s what makes the performance of density based clustering algorithms here somewhat poor. However, if we consider our data as a mixture of vector distributions, and if we find that we have 2 distributions — blue and red, then by comparing the probability that vector A ,B, and C were generated by the blue distribution, their relative similarity (or dissimilarity) comparing to other vectors around them becomes more pronounced. That is so, because its relatively clear that samples B and C were most likely “sampled” from a different distribution than A.

Luckily, this approach is also easy to implement using sklearn’s GaussianMixture model (GMM). When Sklearn’s GMM class is fitted on a dataset, it assumes the data points were generated from a mixture of K Gaussian distributions, and learns a Gaussian Mixture model from the data thereby providing the capability to assign to each sample or vector the Gaussian it mostly probably belong to.

So, for step 1, we use GMM in order to cluster our items and generate item clusters, whereas each cluster corresponds to a Gaussian distribution of item vectors.

I use this simple util function to cluster the items and return the trained model that I need in order to estimate the probabilities later.

The number of K components can be chosen using a variety of methods. I have used the K the minimizes davies_bouldin score for my task.

For stage 2, and this is the important point, for each user vector (that represents the mean of the item vectors the user has listened to), we use the trained model’s predict_prob method to get the probability of its being part of each item cluster.

So, we end up with a dataset of users, whereas each user vector is now represented by a vector of probabilities of its being part of each item cluster. Lets try again to use HDBSCAN to cluster it.

Much better. As expected, our new or transformed data, essentially lumps closer together users whose vector was most likely generated by a certain item cluster or distribution. As mentioned above, this transformation makes similarities or differences between users more “drastic” as now each vector represents a set of probabilities rather than a position in multi-dimensional space.

The results can and should be further examined to see if the clusters makes sense. One problem in this dataset and my implementation is that I didn't take out the most popular artists, which sort of distorted the results, as well as other steps I have skipped for the sake of simplicity, and which also make the results less impressive.

Still, however, we can notice clear topical similarities in the artists that certain user clusters mostly listen to. For example, we notice a cluster of users that share a strong preference for electronic music (cluster 2 in the pic below) or heavy metal rock (cluster 1 in the pic below).

Cluster of users who like electronic music

Cluster of users who like heavy metal rock

IV. Conclusion

User-Item embedding is often hard to cluster when interactions between users and items are more intense and when user preferences vary a lot. The most common approach to such datasets usually involve applying some form of feature transformation in order to make similarities between similar samples more pronounced and differences clearer.

In this post, I have shared another approach or method that may help in addressing such cases. Although it operates in a similar fashion, it can be regarded as a feature generation method rather than a transformation. The method first learns a Gaussian mixture model based on item data, which essentially assumes that the data was generated from a mix of Gaussian distributions. Then, it generates a new feature set for each user vector or record which represents the probability of being sampled from each distribution. In doing so, it essentially, replaces the distance similarity metric with a different metric, which is the probabilities of being part of either distribution.

This method has produced very nice results in a sufficient number of experiments to be worth sharing, but it is far from being seriously thought through or tested. Having said that, it often does a good job and may certainly benefit other domains in which similar difficulties arise.

Hope it will be helpful

notebook can be found here