How to Build a Memory-Based Recommendation System using Python Surprise | by Mate Pocs | Towards Data Science

Ahh, the excruciating pain of our comfortable modern life, the cupcakes all look enticing, but there is no way you can try them all, so which one(s) should you eat? Regardless of the platform, your choices are usually practically limitless, but your resources as consumers are sadly not. Do not worry, recommendation systems come to the rescue!
The framework recommendation system is that we have a set of users and a set of items, and for a given user we want to filter to a subset of items the user in question would enjoy (rate high, buy, watch, etc., depending on the exact problem) with a high probability. Recommendation systems are everywhere, all the content-based very successful tech companies like Netflix, Amazon, Facebook, rely very heavily on sophisticated recommendation systems to increase consumption of their product.
In the particular project that is the basis of this post, I was focusing on the top 100 games from boardgamegeek (as of March 31, 2020), working with 2.3 million individual user ratings I gathered from the site. I primarily used surprise, a Python scikit library that focuses on recommendation systems and its structure closely resembles scikit-learn.
In this post, we are going to discuss memory-based models. We are going to talk about how to import and prepare data, what similarity metrics to use, how to implement the three built-in kNN models, how to apply model validation, and finally make predictions. For details on this project, please refer to my GitHub repository, the work we will cover in this post roughly corresponds to the code in the file 02_modelling_neighbours.ipynb.
Recommendation SystemsData ImportData PreparationModel ParametersKNN ModelsTestingPredictionCross-validationPutting It All TogetherPossible Improvements
Let’s start with a very quick summary of the family of recommendation systems, just so we can place nearest neighbour models.
There are two main recommendation routes you can go with:
Content-based filtering models are based on descriptions of items and the user’s historical preference, and we do not need other users’ opinion to recommend. Example: user enjoys three games that are designed by Vlaada Chvátil, so we recommend a fourth game designed by him.
Collaborative filtering models try to find similarities between items / users through commonly rated /owned items. Example: user enjoys Caverna, from our analysis of the population we know that those users who enjoy Caverna AND know about Feast for Odin tend to enjoy the second game too, so we recommend FfO to the user.
In this project, we are going with collaborative filtering. Within the group of collaborative filtering, the two most well-known distinct approaches are:
Memory-based models calculate the similarities between users / items based on user-item rating pairs.
Model-based models (admittedly, a weird name) use some sort of machine learning algorithm to estimate the ratings. A typical example is singular value decomposition of the user-item ratings matrix.
In this post, we are focusing on memory-based models. So once again, inside recommendation systems is collaborative filtering, and inside collaborative filtering are the memory-based models.
First things first, we need to install the surprise package:
pip install scikit-surprise
Once that is done, you need a dataset, with three variables: user id, item id, and rating. This is important, do not try to pass in the ratings in a user-item ratings matrix format. Start with a data that has 3 columns and the number of rows equal to the total number of individual ratings.
If you just want to practice, feel free to use the dataset on my GitHub. I personally gathered these in a csv file, in three columns, but you can work with other data structures as well, or load in directly from a pandas DataFrame.
In order to import data, you need these classes from the library:
from surprise import Dataset, Reader
Then define the file_path (obviously change to your file):
file_path = './data_input/games_100_summary_w_testuser.csv'
And finally, we create a Reader object, with the following attributes:
line_format: make sure the order matches with your file.
sep: it’s a comma if we are working with a csv.
rating_scale: a tuple with the lowest and highest possible range. It’s important to get this parameter right, otherwise parts of your data will be ignored. If you are using a binary data, for example, did the user like / dislike the item, you can put in (0,1).
reader = Reader(    line_format='user item rating', sep=',', rating_scale = (1,10)    )
To import the data, use the load_from_file method:
data = Dataset.load_from_file(file_path, reader=reader)
And that’s it, you should have your data in a format surprise can work with! You can imagine the data now as a sparse matrix, where the users / items are the rows / columns, and the individual ratings are elements in this matrix. Most of the cells would probably be empty, but that is perfectly OK. In the data I used, I had 2.3m ratings, and about 230k users, which means users on average rated about 10 games out of 100, so the matrix was about 90% empty.
This is where surprise first becomes a bit strange to work with, the process not analogous with let’s say a classifier model in scikit-learn where you have one large matrix, and you can split it into train / validation / test sets, do cross-validation however you see fit, because they are still essentially the same types of data. No, in surprise, there are three different data classes, each with their own distinct usage:
Dataset: used to split into train- and testsets, either directly, or through a cross-validation iterator. This latter means that if you pass a Dataset on as a parameter in cross-validation, it will create many train-test splits.
Trainset: used as a parameter in a model’s fit method.
Testset: used as a parameter in a model’s test method.
In my opinion, surprise is a relatively well-documented library, but there are still some oddities. For example, a Dataset object has a method construct_testset, but other than finding the code in a previous version of the documentation page, there is no explanation for what it does, or what the parameters should be.
I am sticking to the well-documented methods in my project. We are preparing for two different approaches, the purpose of these will become clearer in the following sections.
We are going to use the following, from the model_selection package:
from surprise.model_selection import train_test_split
First, we are splitting the data into trainset and testset, test_size is set to 20%:
trainset, testset = train_test_split(data, test_size=0.2)
Once again, it’s a bit different from how a classifier / regression model would work, the testset contains randomly selected user / item ratings, as opposed to full users / items. One user might have 10 ratings in the data, and 3 of them are now randomly selected for the testset, not to be used in fitting the model. I found this strange when I first noticed, but it makes perfect sense not to completely omit certain users.
The second approach is to use the full data and cross-validate for testing. In that case, we can build a Trainset object using all the ratings with the build_full_trainset method:
trainsetfull = data.build_full_trainset()
You can get the number of items / users with the n_users and n_items methods (same approach works for trainsetfull, because they are the same type of object):
print('Number of users: ', trainset.n_users, '\n')print('Number of items: ', trainset.n_items, '\n')
When surprise creates a Trainset or Testset object, it takes the raw_id’s (the ones that you used in the file you imported), and converts them to so-called inner_id’s (basically a series of integers, starting from 0). You might need to trace back to the original names. Using the items as an example (you can do the same approach with users, just swap iid's with uid's in the code), to get the list of inner_iids, you can use the all_items method. To convert from raw to inner id you can use the to_inner_iid method, and the to_raw_iid to convert back.
An example on how to save a list of inner and raw item id’s:
trainset_iids = list(trainset.all_items())iid_converter = lambda x: trainset.to_raw_iid(x)trainset_raw_iids = list(map(iid_converter, trainset_iids))
And that concludes our data preparation, time to look into some model parameters next!
When we work with kNN — type recommender algorithms, there are 2 hyperparameters we can tune: the k parameter (yes, same k as in the name of the model type), and the similarity option.
The k parameter is fairly straightforward, and analogous to how it works in general k-nearest neighbours models: it is the upper limit of similar items we want the algorithm to consider. For example, if the user rated 20 games, but we set k to 10, when we estimate a rating to a new game, only those 10 games out of 20 that are the closest to the new game will be considered. You can also set min_k, if a user does not have enough ratings, the global average will be used for estimations. As a default, it’s 1.
We mentioned items being close to each other in the previous paragraph, but how do we determine that distance? It is the second hyperparameter, the similarity option, that defines the way to calculate it.
Let’s have a look at the sim_option configuration first. This parameter is a dictionary, with the following keys:
shrinkage: won’t need for basic kNN models, only comes in play with the KNNBaseline model.
user_based: Basically, there are two different routes when you want to estimate similarities. You can either compute how similar each item is to each other item, or do the same with the users. For my project, I used False, considering I had 100 items and 230k users.
min_support: The minimum number of common points under which the similarity will be set to 0. Example: if min_support is 10, and there are two games, only 9 users rated them both, regardless of the ratings, the two games’ similarity will be 0. I did not experiment with this in my project, did not seem to be significant considering the scope of the data, so I used the default 1.
name: The type of formula, to be discussed further below.
All the similarity functions will return a number between 0 and 1 to a specific (i, j) item pair. 1 means the ratings are perfectly aligned, 0 means there is no connection between the two items. In the formulas, rui is the rating user u gave to item i, μi is the average ratings on item i, and Uij is the set of users that rated both items i and j. These are the three similarity metrics in the surprise similarity module:
cosine:
MSD:
where msd(i, j) is:
pearson:
There is no clear good or bad choice, although I rarely saw MSD being used in examples, and in my data, pearson and cosine did perform significantly better. One can see that the pearson formula is basically the mean-centered version of the cosine formula.
An example on how to define a sim_option parameter:
my_sim_option = {    'name':'MSD', 'user_based':False, min_support = 1    }
Now we did all the prep work, we can finally train some models.
There are three variations of basic KNN models in surprise (we are not considering the fourth one, KNNBaseline in this post). They define how rui , the rating user u would give to item i, is estimated in the predictions. The formulas below mostly use notations we have discussed in the previous section, a couple of new ones: σi is the standard deviation of item i, Nuk(i) is the maximum k items from the ones user u rated that are closest to item i.
Without further ado, the formulas:
KNNBasic:
The estimated rating is basically a weighted average of the ratings the user gave to similar items, weighted by the similarities.
KNNWithMeans:
Adjusts KNNBasic formulas with average ratings of items.
KNNWithZScore:
Takes things one step further, also adjusts with the standard deviations of the ratings.
In the example below, we are fitting a KNNWithMeans model, with the three my_ parameters. In my experience, there is hardly any case where you want to go with KNNBasic, if the average ratings of your items are different. You can freely change these parameters as you see fit, and exactly the same parameters are used for all three models. You can change KNNWithMeans to KNNBasic or KNNWithZScore in the code below and it would work just the same.
from surprise import KNNWithMeansmy_k = 15my_min_k = 5my_sim_option = {    'name':'pearson', 'user_based':False,     }algo = KNNWithMeans(    k = my_k, min_k = my_min_k, sim_option = my_sim_option    )algo.fit(trainset)
And with that, our model is fit. Technically, all that happens at this point is that the model calculates the similarity matrix and the mean / standard deviation if you opted for it.
You can request the similarity matrix with the sim method like so:
algo.sim()
It will be in a numpy array format. Unless you want to do some sort of predictions yourself, you will likely not need this matrix.
Once you train your model, it’s time to test, right? Performance metrics are kept in the accuracy module of surprise. There are four metrics available here (RMSE, FCP, MAE, MSE), but as far as I know, the industry standard is the Root Mean Squared Error, so we are only using that one. This is our final math formula for today:
This score roughly tells you how off your estimated ratings are on average from the actual ratings. To get the test score, all you have to do is create a predictions object using the test method on the algorithm that you already fitted:
from surprise import accuracypredictions = algo.test(testset)accuracy.rmse(predictions)
Let’s say that with my data, the RMSE score is 1.2891 on the test data. That would mean the estimated ratings on average are about 1.2891 higher or lower than the actual ratings, on a 1 to 10 scale. Not great, not terrible.
In the previous two sections, we followed a very direct approach: we set aside a test data, trained the model, and then tested its performance. However, if you do multiple runs, it’s probably a better idea to use cross-validation to test your model’s performance and whether it’s overfitting or not.
As mentioned earlier, testing and validating works a bit differently in surprise. You can only cross-validate on your original Dataset object, and can’t set aside a separate test portion for final testing, at least I couldn’t find a way to do it. So my process was essentially this:
cross-validated a number of model types with different parameters,
selected the configuration with the lowest average test RMSE score,
trained that model on the whole Dataset,
used it for predictions.
Let’s discuss a couple of parameters of the cross_validate method:
cv is where we define the type of folding we want the model to use, analogous with how it works in scikit-learn. I am going to use a basic K-Fold by entering an integer, you can read about different ones in the documentation.
n_jobs: the number of folds evaluated in parallel, if you can spare the memory, setting it to -1 will mean all the CPUs are working in parallel. However, when I did that on my local machine, it crashed. I ended up leaving it at the default 1.
In the next segment, we save the results of the cross-validation in the results variable:
from surprise.model_selection import cross_validateresults = cross_validate(    algo = algo, data = data, measures=['RMSE'],     cv=5, return_train_measures=True    )
Please note that it might take a few minutes to run this, testing takes a while, and the cross-validation does that 5 times.
Once it finished, you can dig into the results variable to analyse the performance. For example, to get the average test RMSE:
results['test_rmse'].mean()
Naturally, you would do this for a while, and try out different models, and aim for a low RMSE score. Once you are happy with the performance, and have created an algo model you were satisfied with, it’s time to train the algorithm on the whole dataset. You need this step, because as I mentioned you cannot make predictions from cross-validation. Same code as we did above for the not-full trainset:
algo.fit(trainsetfull)
In the next step, we move on to predictions!
Finally, this is why we did the whole project, right? Two important notes on surprise that might not be aligned with your expectations:
Predictions can only be made on users who are already in the dataset. That’s also a reason why I think it makes sense to train the model on the whole data at the end of the process.
You do not get a list of outputs right from the model, with one call, you can request the estimated rating of one specific item by one specific user. However, there is a workaround, and we will get back to it a bit later.
In order to do one prediction, you can use the raw id’s, so to get the TestUser1’s (a user that has at least min_k number of other ratings in the data) estimated rating of game with id 161936, you need the predict method of your trained algorithm:
algo.predict(uid = 'TestUser1', iid = '161936')
The predict method will return a dictionary like this:
Prediction(uid='TestUser1', iid='161936', r_ui=None, est=6.647051644687803, details={'actual_k': 4, 'was_impossible': False})
r_ui is None, because the user does not have an actual rating for that item. It is the est item that we are ultimately interested in, that is the estimated rating, in our case, we estimate the rating to be 6.647.
So that is all nice, but how do we get the top N recommendations for a user? There is a detailed solution you can find in the documentation, which I’m not going to copy here, these are the basic steps:
Train model on trainsetfull.
Create an “anti testset” with the build_anti_testset method. This is basically the complement of our original dataset. So if a user rated 15 games out of 100, our testset will contain the 85 the user did not rate.
Run the predictions on the anti_testset with the test method (which results in a similar structure to the predict method). With this step, we have an estimated rating for all the user-item rating pairs that was missing from our data.
Sort the estimated ratings for each user, list the N items with highest estimated ratings.
I thought it would make sense to include what we discussed in one continuous block. The approach we are taking in the code below is the cross-validation route, so we test the performance with cross-validation, then fit the model on the whole dataset.
Please note that you would most likely not stop with one cross-validation, and should try different models until you find the best. You also might want to facilitate the top N recommendations linked in the previous section.
There are a number of additional options you have when it comes to working with surprise, and I intend to explore them in future blogposts.
One rather obvious next step would be to explore the model-based approach with the SVD and SVDpp methods. These use matrix factorisation in estimating the ratings. Also, you might have noticed that I was not using GridSearchCV for hyperparameter tuning in this scenario. I found working with cross_validate to be sufficient, considering we only have a few parameters, but when it comes to the more complicated models, you definitely want to use GridSearchCV.
Update: If you are interested in implementing a model-based approach, have a look at my next post in the series, How to Build a Model-Based Recommendation System using Python Surprise.
Another interesting area to explore is the predictions. Sometimes, you just want to run your model on some user ratings, without them being integrated in the underlying database. For example, I collected the data from boardgamegeek, and when I just want to quickly showcase the model to someone, I don’t want those ratings to be mixed in with the “official” ones. It’s also not efficient to re-run the whole model for one user. Now, for the three KNN models we discussed, it’s perfectly possible to make predictions from just the similarity matrix, the mean, and standard deviation. I will dedicate a future post describing the process, or you can just have a look at the recomm_func.py script in my GitHub.
Update: If you want to find about more about this custom prediction code, have a look at My Python Code for Flexible Recommendations.
en.wikipedia.org
The next three links are different documentations of scikit-surprise, I primarily used the first one.