Create a Recommendation System Based on Time-Series Data Using Latent Dirichlet Allocation
By: Alon Agmon
Tags: Time Series Analysis, Data Science, Clustering, NLP, Machine Learning

Create a Recommendation System Based on Time-Series Data Using Latent Dirichlet Allocation

Photo by Lucian Alexe on Unsplash

Intro

Many believe that the main goal of user segmentation and recommendation systems is to predict or better understand what customers want or likely to buy. Sometimes, however, the important question is not what customers are likely to buy, but when are they likely to buy it.

Consider the following examples: (1) An online store wants to send its most dedicated customers a coupon for a product that fits their preferences. However, they also know that customers differ in the time-of-day in which they usually shop —e.g. some prefer to shop early in the morning while others prefer to shop late at night. Thus, they want to know exactly when each customer (or group of customers) is more likely to shop and send the coupon at the appropriate time. (2) A provider of network proxy services needs to efficiently allocate network bandwidth to their users. Ideally, they would like to group users according to their activity hours and allocate them the bandwidth accordingly.

These “business problems” already assume that users or customers want a certain product. The question they pose is when are they most likely to buy or use it. In this post, I would like to share a very simple methodology that can add another tool that might help you answer such questions.

I will assume that you have a simple time series data set in the form of:

Time ('2020-01-05 10:30')

Entity (user, product, game)

Count (views, orders, downloads)

I will show how LDA enables us to (1) segment users into groups or clusters according to the time in which they are more likely to be “active” in quite an insightful way, and (2) let us predict how significantly each user is associated with each group, based on its data. (The same analysis can be also applied to products and cluster them into groups according to the time in which they are more likely to be purchased, etc).

In the sections that follow, I will very briefly introduce LDA and the way it operates, and then dive right into the code.

Latent Dirichlet Allocation (LDA) Clustering — A (Very) Short Intro

LDA is most commonly used for topic modeling in NLP contexts, and that is indeed a good example use case to grasp its value. Topic modeling algorithms usually start with a “corpus” or a group of “documents” that consist of “words”, and try to use the observed mixture of words and the frequency in which they show together in order to find latent patterns or topics within the documents.

For example, suppose that I have 3 documents: (1) “My dog ate my lunch”; (2) “I just love eating fruits”; (3) “I am allergic to dogs”. A topic modeling algorithm should be able to identify at least 2 latent topics or themes within this “corpus”: a topic that deals with dogs and a topic that deals with food or eating. Some topic modelling algorithms (such as LDA) would also be able to associate a document with more than one topic and tell us which topic is more dominant in the document. For example, these algorithms would be able to tell us that sentence 1 (“My dog ate my lunch”) is both about dogs and about food or eating, and also that its more about dogs than it is about food.

So how does LDA do this?

LDA makes at least 2 important working assumptions :

(1) Each topic is a probability distribution of words. Simply put, each topic is associated with certain words that occur in certain probabilities in a document that deals with the topic. One of the main outputs of an LDA model is a term-topic matrix that shows the probability of each word occurring in a certain topic. This matrix allows us to find semantic themes in each topic. For example, if WORD 0 and WORD 1 in the matrix below relate to animals then we can say that TOPIC3 (or its theme) is animals because words that relate to animals are very probable to occur within it.

term-topic Matrix (or feature-cluster matrix)

(2) Each document has a “topic recipe” or a probability distribution of topics. This means, for example, that running sentence 1 (“My dog ate my lunch”) through the model will tell us that the document is 80% about food and eating and 20% about dogs, based on the words it contains.

Topic Recipe

sentence | topic-1 | topic-2 | topic-3 |

s1 ____| 80% _____| 20% ____| 0% ____ |

s2 ____| 50% _____| 30% ____| 20% ____ |

s3 ____| 10% _____| 20% ____| 70% ____ |

When we fit LDA on a certain corpus or group of documents, then it essentially learns which probability distribution of words per topic can best explain (or generate) the distribution of words in each document in our corpus. More accurately, a trained LDA model can reproduce the corpus it was trained on (or at least something semantically similar to it), given the topic recipe. And, more importantly, a trained or fitted LDA model can predict the “topic recipe” of documents unseen by the model when it was trained.

Photo by Ugur Akdemir on Unsplash

What about time-series data?

A trained LDA model allows us to estimate the “topic recipe” of any vectorized document . In the context of NLP, where “documents” are chunks of text or BOW vectors of words, the document’s topic recipe can help us figure out what are the dominant themes of the document or which theme is more dominant than others.

However, at the end of the day, LDA is a clustering algorithm —that is, it estimates a set of vectors with N features and finds latent patterns in the values assigned to these features in each vector. In other words, if we think about topics as clusters and about a topic recipe of a document as a cluster recipe of a vector, telling us how strongly the vector is “associated” with each of the clusters, then its obvious that we can make LDA work for many business problems outside the context of NLP.

This may still seem not entirely related to the domain of time-series analysis, so let me just jump right into an example that I’ll be using throughout this post which will illustrate this more clearly.

An Example: Users and TV Shows

Suppose that you have a dataset of ‘customers’ which records the number of TV shows that have been watched by each customer each hour over a period of 24 hours. Each vector in our dataset thus looks something like this:

[customer, # of shows watched on 1AM, # of shows watched on 2AM….]

Our task is to find an efficient way to predict when is the best time to offer a given customer the option to watch a new TV show. Obviously, there is more than one way to approach this. The approach that I will take here is to try and segment customers into groups according to their TV-watching hours and predict the best time to offer a customer a new TV show according to the preferences or activity of the group they are most dominantly associated with.

How can LDA help us out here?

If we treat each hour of day as a feature with a value equals to the number of TV shows watched by the customer, on average, then estimating LDA on this data will enable us to segment our customers to groups according to their TV show watching hours.

Customers vs. Hours (value = # of TV shows)

More importantly, we will be able to produce a feature-cluster matrix (or hour-group matrix) that will tell us which hour is more significant in which cluster (and how more significant it is), in addition to providing us a cluster recipe for each customer that will tell us how strongly the customer is “associated” with each of the groups. Note that while this task can be also accomplished using others clustering algorithm (such as KMeans), the ability to easily produce a feature-cluster matrix that can quantify the significance of the association of each feature to a cluster is a stark advantage of using LDA (though there are more, but this discussion is beyond my scope here).

Let’s Code

We start with this data set:

The first thing we ought to do is to transform the data to the time series format we want to work with.

We fix the time column on our DataFrame

df_raw['ts'] = pd.to_datetime(df_raw['ts'])

2. We transpose it using the pivot_table method in order to get a row for each customer with every hour (or time bin) as a feature/column

df_pivot = pd.pivot_table(

df_raw,

index='customerid',

columns='ts',

values='shows',

fill_value=0)

3. Finally, we use Panda’s GroupBy to get the mean for each hour

df_avg = df_pivot.T.groupby(df_pivot.T.index.hour).mean().T

Now, we have our data in the right format

When we look on the mean views per hour of our entire dataset we indeed see a somewhat expected curve in which more TV shows are viewed during the prime day hours and less during late night hours. However, as we shall shortly see, using this data as an indicator for when is the best time to offer a new TV show overlooks many customers with preferences that diverge from the mean.

Mean TV shows watched per hour

The simple magic happens in these 3 lines of code

from sklearn.decomposition import LatentDirichletAllocation as LDA lda = LDA(n_components=3, learning_method='batch') lda = lda.fit(df_avg)

We assume that our customers’ viewing preferences can be divided into 3 groups, and fit() the LDA object on our data. And, that’s it. All we need to do now is extract the latent insights in our data.

Extracting Insights

The first and most important insight that LDA produces is a term-topic matrix, or an hour-cluster matrix, that will tell us which viewing hours are more dominant with each cluster. This matrix is usually helpful as a sanity check, to examine whether the clustering makes sense or whether further adjustments are needed.

*Note that the first line will produce this matrix but it is more readable when normalized, which is what the lines that follow do.

#Create the hour-cluster / term-topic matrix

df_comp = pd.DataFrame(lda.components_.T, index=df_avg.columns) #Normalize it norm_components = \

lda.components_ / lda.components_.sum(axis=1).reshape(-1, 1) df_comp_norm = pd.DataFrame(norm_components.T, index=df_avg.columns) df_comp_norm * 100

And here is a more refined version:

We can clearly notice groups with preferences that “make sense”. For example, we can say that cluster 0 describes customers that mostly watch TV shows during the night / late evening hours — there is a chance of 40% (20% + 20%) that the time they will watch a TV show each day is between 8PM and midnight. We also see a group that seems to consist of “daily” watchers (cluster 1) who watches the most during prime day hours 8AM — 4PM.

I will shortly return to this, but once the hours probability distribution within our clusters makes sense, we can use the transform() method which will produce the vector-cluster (or customer-cluster) matrix that will show the extent to which each customer is associated with each of the clusters.

We saw earlier that the prime hours associated with Cluster 0 group are evening-night hours, whereas the prime hours associated with Cluster 1 group are noon-afternoon hours. Accordingly, we can say that the first customer ([0.79, 0.14, 0.06]) mostly watches TV shows on evening-night hours, whereas the second ([0.36, 0.62. 0.01]) mostly watches TV shows during the day. If we could pick just one time frame to present new shows to our customers, we would probably pick the evening/night hours because although the second customer mostly watches their TV during the day, there is still a 36% chance that they will watch a late night show.

We can illustrate this by assigning each customer to its most dominant cluster and then plotting the mean for each cluster.

mean value per feature

We can also use this to evaluate the size of each group. That will show us, for example, that the group that is associated with somewhat abnormal activity hours (cluster 2) is some kind of outlier that needs to be further explored (e.g. perhaps it is related to users from a different time zone?)

At the beginning of the post, I mentioned 2 main questions that we need to answer, or 2 main insights that we want LDA to help us with: (1) We want to know the prime hours for each group (the hours in which they watch the most) in order to present the members of each group the option to watch a new TV show when they are more likely to watch it. (2) Given a customer’s TV watching activity, we want to know how strongly the customer is associated with which group.

One approach we can use to achieve the first task rather simply, is by selecting the mean watching time for each group or cluster, and then find the hours whose values exceed the mean. As the example below shows, the prime hours for the first group (the night watchers) is 4PM to midnight.

As for the second task (find the best hours to present a TV show to a particular customer), all we have to do is call the transform() method using the watching hours vectors of the customer we want to predict.

customer 120

Our model tells us that customer 120 mostly likes to watch their TV during the night, though they also spend some time watching TV during normal day hours. In other words, although they watch more TV at night time, it might also be worthwhile to offer them a new TV show to watch during the day.We can actually confirm this by examining their actual watching hours.

df_avg.iloc[customerid,:-1].plot.bar()

customer 120

Indeed, we can see that the best time to show customer 120 a new TV show is between 10PM and 2AM though they do spend a decent time watching TV during the day.

Summary

To sum, we have quite a few tools to cluster users based on their time-series activity data. I have tried to show a rather simple approach that can certainly expand your toolset when analyzing such problems and easily produce valuable insights.

Hope this was helpful!

You can find the full notebook here

Some nice links about LDA: