A Complete Guide To Recommender Systems — Tutorial with Sklearn, Surprise, Keras, Recommenders | by Pathairush Seeda | Towards Data Science

We all know that the recommender system plays a vital role in many industries ranging from retail, E-commerce, and entertainment to food delivery, etc. It heavily uplifts the user experience on any platform. Imagine that you scroll the marketplace feed repeatedly, and you are so satisfied with all the recommended stuff in your hands even though you may not want it. This component is a de-facto standard for any business.
For myself, I entered the data science field with the impression in the recommendation system of Spotify as well. I was surprised at how precise the recommendation of Spotify is. I explore the new trends and genres of music I've never heard before and exploit what kinds of songs I love. This inspiration drives me to write the article.
What you can expect in this article:
Outline — An introduction to the outlook of the recommendation system
Implementation — The explanation of how to implement each kind of recommendation system.
The following implementations are provided:
Popular-based with IMDB weighted rating
Content-based with the genres feature
Collaborative filtering — Memory-based — Affinity matrix
Collaborative filtering — Memory-based — TruncatedSVD (Sklearn)
Collaborative filtering — Model-based — Funk MF (Surprise)
Collaborative filtering — Model-based — GMF (Keras)
Collaborative filtering — Model-based — NCF (Recommender)
Other states of the art algorithms
Also, the hands-on tutorial notebook is provided at the end of the article.
Today, we will drive into various kinds of recommender systems, and we will provide you with the hands-on tutorial code and explanation for each section. We hardly found the complete guide of both descriptions and hands-on tutorials. So this article will consolidate both aspects together to provide you with a one-stop service for starting the recommendation system implementation.
The above figure shows the high-level overview of the recommender system. It looks like it doesn't have many kinds of recommender engines. However, there are many variations within each recommendation based. We will walk you through some algorithms and provide you with further resources to explore. Nowadays, this research field still grows rapidly. Thus, It won't be that easy to capture all the state-of-the-art techniques within this single article.
Now, let me explain to you a basic description of each building box above.
1. Popular-based — This is the baseline performance and the most intuitive recommendation that we can find anywhere. Examples are the IMDB top-rated movies, Top 10 in your country today in Netflix, etc. These recommendations can be found when you are a new joiner and the provider doesn't have enough information about you. So it would be a safe bet to recommend to you what others like.
❗ Limitation: All users get the same recommendation set. It's not personalized.
2. Content-based — If you would like to improve the popular-based method, we can incorporate each item's detail to recommend more relevant to the user's desires. For example, you may see the following figure on Netflix. It provides you with a list of items because you like some specific movie. The movie would be relevant or similar to what users like in some aspects such as genres, sub-genres, channel, country, etc. The quality of the recommendation depends on how rich your feature set is.
❗ Limitation: The recommendation will be limited to what users liked, watched, interacted with before. It doesn't give users a chance to explore a new area they’ve never been to before. Also, all users who like item X will receive the same recommendation set.
3. Collaborative filtering — Now, what if you have prior information about the user and the item the user interacted with before. How can you come up with a more sophisticated recommendation engine? This is where collaborative filtering comes to play. Collaborative filtering recommends the set of items based on what is called the user-item interaction matrix. Here is how the user-item interaction matrix look likes.
The number in each cell represents the explicit or implicit rating that the user gives to the specific items.
To elaborate on the explicit, implicit rating.
The explicit rating is something we directly ask the user to rate the satisfaction score for the specific item. For example, if we assume the number is the satisfaction rating the user gives to the particular item, ranging from 1–5 scales. It means that user 1 likes Item#1 with a satisfaction score of 3.
On the other hand, the implicit rating is the rating that we indirectly derive from user behavior. The example could be the minutes that their user watched the movie or how many clicks the user browsed through the product page.
The choice of how to structure the user-item interaction matrix depends on the objective of your business. Given that the above matrix now we would like to create a personalized recommendation for each user.
There are two major ways to do it in the collaborative filtering method.
3.1. User-based collaborative filtering — This technique will personalize our recommendation based on the similar group of users we derived from the above user-item interaction matrix. The below figure shows you how we came up with the set of recommendations for user#1. You can see that for each user, the set of recommendations will change based on the group of similar users, and the group of similar users will vary based on how user#1 interacts with each item.
⭐️ Notice: This is the recommendation set that is personalized for each user. Every user will get their recommendations based on what item they interacted with in the past, and this will lead to a different set of recommendations for each user unless they are like all the same set of products.
❗ Limitation: we need prior information about the user before we can derive the similarity for user#1 and others. If you don't have any information about user#1, you can't come up with the prediction because you can't calculate the similarity between users. This problem is called the Cold-Start problem. We can fix this problem by providing the new user with the list of Popular-based recommendations first and let them score how much they like each famous movie first until we have sufficient information.
3.2. Item-based collaborative filtering — Instead of finding a group of similar users, we will find the group of similar items to the one that user #1 just likes and recommend it to user#1. This method will be similar to the popular-based or content-based recommendation. However, the difference is how to come up with a set of similar items. Here we use the user-item interaction matrix rather than the rating or genres like the methods mentioned above.
❗ Limitation: You need prior information about the new item and the other items, as well as the user-based collaborative filtering. Otherwise, you can't be able to derive similar groups of items to recommend. We can fix this problem by providing the Content-based recommendation for the new item first until they have sufficient information.
4. Hybrid — We see that each method has its strength. It would be best if we can combine all those strengths and provide a better recommendation. This idea leads us to another improvement of the recommendation, which is the hybrid method. For example, we can combine the content-based and item-based collaborative filtering recommendations together to leverage both domain features (genres and user-item interaction).
Now, we have completed a brief introduction to the recommender system. The next topic we will discuss is the variation of all the above methods.
How can we map those high-level concepts to the actual algorithm?
How many kinds of recommendation algorithms can we use to derive the best set of recommendations?
How can we evaluate the performance of each method? How do we compare it on the same basis?
Let's start.
The popular-based implementation can be done quickly. You can start with the item with the highest average rating among all users, the item with the highest vote count, or the number of members watched.
To make it more sophisticated, we introduce you to the popular-based method with IMDB weighted rating.
Regarding the IMDB system, there are metrics called weighted rating systems used to score the rating of each movie.
Here is the formula of the weighted rating score.
WR = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C
Where
R is the average rating for the item.
v is the number of votes for the item.
m is the minimum votes required to be listed in the popular items(defined by > percentile 80 of total votes)
C is the average rating across the whole dataset.
With the average rating for each item, we adjust the score based on the number of votes received. We also filter the groups of items based on the number of votes to ensure that the score is adjusted among the suitable candidate.
With the provided code snippet, we generate the list of popular items. Let’s see the top 10 popular items based on the IMDB weighted rating. Here we use the synthesized data as our example in the calculation.
Now, we can bring the name in the chart above to the UI of our application like the one below.
We can add pre-defined features such as time and location to make it more relevant to users' preferences. As you can see in the above figure, Netflix also adds the Time (Today) and Location (Your country) to differentiate the top 10 populated-based recommendations.
For the content-based, we will use the genres feature of each item to recommend the relevant or similar item to the user.
Based on our dataset, we have genre features like action, comedy, drama, etc. We will do the one-hot encoding of the genre column and find the similarity between each item. The last step is to rank the similarity score from highest to lowest and select the set of items based on the number of recommendations we would like to offer.
Here is the snippet code to make the content-based recommendation.
The related items data frame looks like this.
Let's look up item id 99. You can see that the top-recommended sets are towards the Romance and Action genre, which aligns with the content-based recommendation concept.
To link you back to the interface of the content-based recommendation, we can fill the following UI template with our list of recommendations.
There are two main approaches to collaborative filtering.
Memory-based
Model-based
It seems like there are not that many approaches we can build the model. However, the variation inside each approach is limitless. You will see more detail as you go through this article. Let's start!
The memory-based approach is so simple that It calculates the similarity matrix directly from the user-item matrix.
Let's look at the user-item interaction matrix. The first thing we need to do with collaborative filtering is to find the similarity between the user or item.
Pearson correlation — The most well-known similarity metric for the linear relation is person correlation. It measures how similar two samples are based on the direction of how the value changes.Cosine similarity — As the name mentioned, It measures the cosine angle of the two vectors in the multi-dimensional space. Two things can be similar together in terms of direction rather than magnitude.Other similarity scores — There are many other similarity scores that we can come up with. Here is the example A hybrid user-based collaborative filtering algorithm with topic model [1]. They provide a way to compute the similarity metric with the Time-Decay Latent Dirichlet Allocation, which is an excellent example of how we can play with this part of the recommendation system.
Pearson correlation — The most well-known similarity metric for the linear relation is person correlation. It measures how similar two samples are based on the direction of how the value changes.
Cosine similarity — As the name mentioned, It measures the cosine angle of the two vectors in the multi-dimensional space. Two things can be similar together in terms of direction rather than magnitude.
Other similarity scores — There are many other similarity scores that we can come up with. Here is the example A hybrid user-based collaborative filtering algorithm with topic model [1]. They provide a way to compute the similarity metric with the Time-Decay Latent Dirichlet Allocation, which is an excellent example of how we can play with this part of the recommendation system.
You can select any similarity metric you prefer. With the similarity metric of your choice, we can calculate the similarity between user or item and then recommend based on the following step.
User-based implementation
We find the group of similar users (the group size is arbitrary) based on your similarity metric choice.We average the rating of each item based on the group of similar usersRank the item based on the descending average rating, and recommend the target user with the item they never interacted with before.
We find the group of similar users (the group size is arbitrary) based on your similarity metric choice.
We average the rating of each item based on the group of similar users
Rank the item based on the descending average rating, and recommend the target user with the item they never interacted with before.
Item-based implementation
We find the group of similar items based on your similarity metric choice.Select up to the top-k most similar item to recommend.
We find the group of similar items based on your similarity metric choice.
Select up to the top-k most similar item to recommend.
⭐️ Notice: You can see that we can derive the recommendation set without learning parameters as we did in the other machine learning models. This is where the name of memory-based came from. We create the engine that remembers what users like and don't like then we retrieve the result based on the similarity of those interactions—no need for inferencing anything.
❗ Limitation: because the idea of the approach is to memorize every interaction between user and item, the problem that will happen here is the scalability of the engine. In reality, the imbalance between the number of users and items makes the user-item matrix very sparse, leading to the poor generalization of the predicted result.
Another point to consider, imagine that you have a new user and item, and you need to make a new recommendation for those new users. You have to fit the model once again by adding those new users and compute similarities between each pair of users. Also, the user and item size in the real application are gigantic, leading to more sparsity of the user-item interaction metrics. This execution plan will lead to infinite resource consumption within a short time.
Now, instead of direct computation with the user-item interaction matrix. We will decompose the user-item interaction matrix into the latent factors matrix representing the lower-dimensional space that is more useful.
Let me briefly walk you to the mathematical concept behind this idea. The idea of decomposing is we believe that the observed user-item rating matrix is constructed from the underlying user and item latent factor matrix.
Suppose we can extract the best underlying latent factor matrix that minimizing the loss between the reconstructed matrix and the original matrix. Then we can use the inner product of the user and item latent factor matrix for inferencing an unobserved rating.
We call this concept and approach Matrix Factorization. There are several kinds of matrix factorization techniques, and each of them provides a different set of results, leading to different recommendations.
This is the place where classic methods like Singular Value Decomposition, Principal Component Analysis, or advanced cutting edge technology like Deep learning show up. The research of this field is still moving forward rapidly.
This is where you can have a lot of room to play with. As the name suggested, we will incorporate some kinds of models to use as a recommendation engine for prediction.
We learn from the memory-based approach that is directly computing the similarity of the user and item is not practical in terms of scalability.
⭐️ Notice: The key important that differs between the model-based and memory-based methods is the model-based involves building a model based on the dataset of ratings. To be more precise, we extract the data from the user-item interaction matrix and use that as a model to make recommendations. This solves the scalability problem of the memory-based approach and hence makes the real-world implementation easier.
Here we will explain three kinds of matrix factorization that we implement in the hands-on tutorial notebook. There is a basic technique we can use the create a simple go-to recommendation engine.
However, If you would like to take a look a the state-of-the-art technique, I will also attach the resources I found interesting for your further exploration as well.
The first approach we will walk you through here is the matrix factorization with the truncatedSVD with the sklearn library.
TruncatedSVD is a variant of the Singular Value Decomposition that calculates only the K largest singular value (n_components). Also, It applies the linear dimensionality reduction and works well with the sparse matrix like the user-item matrix.
We aim to decompose the user-item matrix into these latent factors. The value of each cell will be the estimated value that satisfies the optimization constraint (SVD assumption). An example of another matrix factorization is Non-negative matrix factorization (NMF).
Here is the sample snippet code of how to apply the truncatedSVD to the user-item matrix in python
It seems like a similar version for this approach, but we have added the decomposition step into account. This will make the recommendation more robust and reduce the memory consumption from the large size of the user-item interaction matrix. However, when we have a new user or item, we still need to refit the user-item interaction matrix before making the prediction.
After that, we can implement the step as the same as memory-based approaches to create user-based or item-based recommendations.
⭐️ Notice: Now, the recommendation would be based on some latent factors that we cannot explain directly compared to the explicit or implicit rating from the memory-based approached.
❗ Limitation: The scalability is still a problem for this algorithm, even if we reduce the size of a matrix with the decomposition method. Also, the explainability would be a problem too. We don't know how to describe each latent factor in terms of human interpretation.
The algorithm is initiated by Funk, Simon [2]. It holds the concept of matrix factorization, which reduces the user-item interaction into the lower dimensional space latent matrix. However, there is no dimensionality reduction technique like SVD or PCA applied under the hood.
⭐️ Notice: even if the method name in the Surprise package is SVD, it's a different approach and is not related to the Singular Value Decomposition method. The Funk MF is just an SVD-like algorithm.
The objective of FunkFM is to estimate the latent factor matrix and the bias termed minimizing the loss between the original explicit rating and the reconstructed prediction rating. Therefore, that is why we called this kind of approach model-based.
To make a predicted rating, we can multiply the estimated latent factors matrix and add the bias term for the user or item like the below figure.
Where
the u is denoting user index, and i is denoting the item index.
r(hat)ui is the predicted rating of the user u on the item i
mu is the mean of all ratings
b is the bias term (baseline) for the user or item
qi is the item latent factor matrix
pu is the user latent factor matrix
If we don't know the information about the user, then the term bu and pu will be assumed to be zero. Thus, the predicted rating of the new user will be the mean of all ratings plus the bi term, which means if we don't know the user, we will recommend them with the product with a high baseline term that we learned from the data.
The same applies to the item termsbi and qi . If it's the new user and new item, we will have the predicted rating equal to the mean of all ratings.
To give you a better sense of what to optimize for, here is the loss function implemented in the Surprise library.
Where
lambda is the regularized hyperparameter we can tune.
We can also add the regularization on both estimated latent factors and the bias term and adjust the number of latent factors in the model.
All the learning parameters will be randomly initialized and updated using the stochastic gradient descent (SGD) method. The fitted model will be used for predicting the rating of the specific user on the specific item.
❗ Limitation: as you can see in the rating prediction, this model only takes into account the explicit rating (a true rating that the user gives to the item), and it doesn't care about the implicit rating (the number of clicks, the time spent on the item, etc.). There is an improvement about this Limitation as well. For example, you can check out the SVD++ algorithms.
Here is the sample snippet code of how to apply the funk MF to the user-item matrix in python
⭐️ Notice: The name of this method is not universal. It is only valid in the context of Neural Collaborative Filtering [3].
For this implementation, when I started to learn how deep learning works with the recommender system, I found this tutorial on this Keras example. With a short and precise code snippet, it helps me a lot to understand how to structure the neural network architecture for the recommendation engine.
I also found out that the implementation is based on the part of the following research from Neural Collaborative Filtering [3]. In section 3.2, the paper shows how the matrix factorization can be treated as a special case of the neural collaborative filtering (NCF) framework.
They started with the idea that the embedding layers that dense the sparse input user and item vector (user-item interaction matrix) can be seen as a latent factor matrix in the normal matrix factorization process.
With the mentioned idea in mind, this is how they provide the prediction.
Where
y(hat)ui is the prediction of the user u on the item i
a(out) is the activation function at the final layer of the neural network
h^T is the edge weight of the output layers
pu and qi can be seen as the latent vector resulting from the embedding layer here.
circle(.) is element-wise product operation
From how the prediction is derived, if we use the identity function as an activation function a(out) and use the uniform vector of one for edge weights of the output layer h^T . We can prove that the matrix factorization is the special case of the NCF framework, which is the prediction that came from the inner product of the latent factors matrix.
However, for the GMF mentioned here, they replace the a(out) with the sigmoid function and learn h^T from the input data that minimizes the log loss.
To summarize, the architecture of the GMF model is the partial red line in the below figure. It's only part of the whole NCF model that we will discuss further in the next topic.
With that said, let's see how we implement this model in the Keras. Here we also add the variation of the neural network architecture to predict the rating instead of value between 0 and 1 as the reference paper proposed.
Now we see how to integrate and build the recommender system with the Keras library. This unlocks the possibility to make your architecture for the recommendation engine with the deep learning architecture. It's another kind of recommendation engine that we can tweak and play with compared to the provided package like Surprise.
Last but not least, let's see the full architecture and the power of the NCF algorithms that we mentioned earlier.
You can see that the full NCF architecture has a multi-layer perception (MLP) part. This proposed idea incorporates and activates how the model can estimate the latent factors matrix with the non-linear function. The idea is that due to the complexity of the user-item interaction matrix, only the linear product of the previous matrix factorization technique is not enough to retrieve useful information. Therefore, the idea to add the MLP part to help capture the pattern in the data is proposed.
Based on the preliminary comparison on the movielens@100k data set in the Recommender library, The NCF algorithm outperforms the traditional Funk MF (SVD-like algorithm in Surprise package). This evidence indicates how important the deep learning approach is in the recommendation system.
Let's see the last snippet code for a hands-on tutorial with the Recommender library.
When I learned and researched recommender systems, it led me to the Recommender library provided by Microsoft. This library elevates my perspective and understanding of the recommendation system to the next level.
At first glance at the Recommender library, I was overwhelmed with many unseen approaches. Some approaches seem to be state of the art for the recommendation engine nowadays. But after I spent some time with Github, documentation, and the provided examples. I realized that all of the contents are very rich and useful.
If you come to this part of the article, you will have enough familiarity with the broad overview of the recommendation system. However, the sky is limitless. You can go further in this path with the stuff in the Recommender library. It would take some time to go deep dive into the various detailed algorithms, but we think it's really worth your time if you need to excel in this field of recommendation systems.
I will provide you with some interesting resources for reading next in the following section.
Here are the interesting resources that I think you should take a look
Bilateral Variational Autoencoder (BiVAE)
Bayesian Personalized Ranking (BPR)
Light Graph Convolutional Network (LigthGCN)
Transformaer4Rec provided by Nvidia-Merlin
List of Recommender System
Here we come to the end of this article. I hope all the material and explanations are useful enough to start your journey in the recommendation system domain.
It's difficult for me at the beginning to gather information on the internet without the proper direction. Thus, I believe this guide will help you start from the basics of the recommender system and provide you a further reading and where to explore for future endeavors.
There are a lot of things going on in this field. Hence, you have to continually keep up with the technology to leverage all your data and provide the best satisfaction set of recommendations to all of your users!
You can find all of the following codes here in this Colab notebook. The notebook is still under development, and I will keep it update as I learn more about the recommendation system.
colab.research.google.com
[1] Na, L., Ming-xia, L., Hai-yang, Q. et al. A hybrid user-based collaborative filtering algorithm with topic model. Appl Intell 51, 7946–7959 (2021).
[2] Funk, Simon. "Netflix Update: Try This at Home" (2006)
[3] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural Collaborative Filtering (2017), In Proceedings of the 26th International Conference on World Wide Web (WWW '17).