Perfecting Recommendation Pipeline with Advanced Statistical and Algebraic Computations
By: The Immaculate Blogger
Tags: Recommendation System, Svd, Collaborative Filtering, Cosine Similarity, Data Science

This post is an extension to my last blog that described the most lucid pipeline of general recommendation systems through a promising demonstration with the most popular MovieLens data set. However, it suffers from some inherent fallacies as it takes into account only the similarity aspect and rules out the other relevant considerations that may play significant role in establishing more accurate correlation between the actual versus to be recommended items. This goes without saying that formulations hereafter is quite generic and can be leveraged unequivocally to recommend movies, musics, books, dates(dating apps), healthcare and other recommendable products or services. Commercially employed recommendation systems are undoubtedly very good at mining all potential features that data corpus could offer and that too optimally.

With the ambition to decipher and discern the technical crux behind such complex yet accurate recommendation system, I have endeavored to present a better version of pipeline incorporating advanced dimensions of features impacting the results.

Standard Methods

Recommendation Systems usually rely on larger data sets and specifically need to be organized in a particular fashion before cool stuff burgeons. I have ensured to capture an intensive walk through process on creating a recommendation system using Python with the same Movie Lens Data Set. Two most common types of recommender systems are Content-Based and Collaborative Filtering (CF).

Collaborative filtering produces recommendations based on the knowledge of users’ attitude to items, that is it uses the “wisdom of the crowd” to recommend items.

produces recommendations based on the knowledge of users’ attitude to items, that is it uses the “wisdom of the crowd” to recommend items. Content-based recommender systems focus on the attributes of the items and give us recommendations based on the similarity between them.

Collaborative Filtering

In general, Collaborative filtering (CF) is more commonly used than content-based systems because it usually gives better results and is relatively easy to understand (from an overall implementation perspective). The algorithm has the ability to do feature learning on its own, which means that it can start to learn for itself what features to use.

CF can be further divided into Memory-Based Collaborative Filtering and Model-Based Collaborative filtering.

In this tutorial, we will implement Model-Based CF by using singular value decomposition (SVD) and Memory-Based CF by computing cosine similarity.

The Data

We will use famous MovieLens data set, which is one of the most common data sets used when implementing and testing recommender engines. It contains 100k movie ratings from 943 users and a selection of 1682 movies.

We can download the dataset here or just simply use the u.data file that is already included in my github source code.

Lets Get Started

Importing standard libraries and data overview.

2. Merging the two data frames.

3. Train Test Split.

Memory-Based Collaborative Filtering

Memory-Based Collaborative Filtering approaches can be divided into two main sections: user-item filtering and item-item filtering.

A user-item filtering will take a particular user, find users that are similar to that user based on similarity of ratings, and recommend items that those similar users liked.

In contrast, item-item filtering will take an item, find users who liked that item, and find other items that those users or similar users also liked. It takes items and outputs other items as recommendations.

Item-Item Collaborative Filtering: “Users who liked this item also liked …”

User-Item Collaborative Filtering: “Users who are similar to you also liked …”

In both cases, we create a user-item matrix which built from the entire dataset. Since we have split the data into testing and training we will need to create two [943 x 1682] matrices (all users by all movies). The training matrix contains 75% of the ratings and the testing matrix contains 25% of the ratings. After we have built the user-item matrix we calculate the similarity and create a similarity matrix.

The similarity values between items in Item-Item Collaborative Filtering are measured by observing all the users who have rated both items.

For User-Item Collaborative Filtering the similarity values between users are measured by observing all the items that are rated by both users.

A distance metric commonly used in recommender systems is cosine similarity, where the ratings are seen as vectors in n -dimensional space and the similarity is calculated based on the angle between these vectors. Cosine similiarity for users a* and *m can be calculated using the formula below.

To calculate similarity between items m* and *b we use the formula:

Algebraic Computations

Our first step will be to create the user-item matrix. Since we have both testing and training data we need to create two matrices.

Predictions

Next step is to make predictions. We have already created similarity matrices: user_similarity and item_similarity and therefore we can make a prediction by applying following formula for user-based CF:

We can look at the similarity between users k* and *a as weights that are multiplied by the ratings of a similar user a (corrected for the average rating of that user). We will need to normalize it so that the ratings stay between 1 and 5 and, as a final step, sum the average ratings for the user that we are trying to predict.

The idea here is that some users may tend always to give high or low ratings to all movies. The relative difference in the ratings that these users give is more important than the absolute values. To give an example: suppose, user k* gives 4 stars to his favourite movies and 3 stars to all other good movies. Suppose now that another user *t rates movies that he/she likes with 5 stars, and the movies he/she fell asleep over with 3 stars. These two users could have a very similar taste but treat the rating system differently.

When making a prediction for item-based CF we don’t need to correct for users average rating since query user itself is used to do predictions.

Evaluation

There are many evaluation metrics but one of the most popular metric used to evaluate accuracy of predicted ratings is Root Mean Squared Error (RMSE).

We can use the mean_square_error (MSE) function from sklearn , where the RMSE is just the square root of MSE. To read more about different evaluation metrics we can take a look at this article.

Since we only want to consider predicted ratings that are in the test dataset, we filter out all other elements in the prediction matrix with prediction[ground_truth.nonzero()] .

Memory-based algorithms are easy to implement and produce reasonable prediction quality. The drawback of memory-based CF is that it doesn’t scale to real-world scenarios and doesn’t address the well-known cold-start problem, that is when new user or new item enters the system. Model-based CF methods are scalable and can deal with higher sparsity level than memory-based models, but also suffer when new users or items that don’t have any ratings enter the system. I would like to thank Ethan Rosenthal for his post about Memory-Based Collaborative Filtering.

Model-based Collaborative Filtering

Model-based Collaborative Filtering is based on matrix factorization (MF) which has received greater exposure, mainly as an unsupervised learning method for latent variable decomposition and dimensionality reduction. Matrix factorization is widely used for recommender systems where it can deal better with scalability and sparsity than Memory-based CF. The goal of MF is to learn the latent preferences of users and the latent attributes of items from known ratings (learn features that describe the characteristics of ratings) to then predict the unknown ratings through the dot product of the latent features of users and items. When we have a very sparse matrix, with a lot of dimensions, by doing matrix factorization we can restructure the user-item matrix into low-rank structure, and we can represent the matrix by the multiplication of two low-rank matrices, where the rows contain the latent vector. We fit this matrix to approximate our original matrix, as closely as possible, by multiplying the low-rank matrices together, which fills in the entries missing in the original matrix.

Let’s calculate the sparsity level of MovieLens dataset:

To give an example of the learned latent preferences of the users and items: let’s say for the MovieLens dataset we have the following information: (user id, age, location, gender, movie id, director, actor, language, year, rating). By applying matrix factorization the model learns that important user features are age group (under 10, 10–18, 18–30, 30–90), location and gender, and for movie features it learns that decade, director and actor are most important. Now if we look into the information we have stored, there is no such feature as the decade, but the model can learn on its own. The important aspect is that the CF model only uses data (user_id, movie_id, rating) to learn the latent features. If there is little data available model-based CF model will predict poorly, since it will be more difficult to learn the latent features.

Models that use both ratings and content features are called Hybrid Recommender Systems where both Collaborative Filtering and Content-based Models are combined. Hybrid recommender systems usually show higher accuracy than Collaborative Filtering or Content-based Models on their own: they are capable to address the cold-start problem better since if we don’t have any ratings for a user or an item we could use the metadata from the user or item to make a prediction.

SVD

A well-known matrix factorization method is Singular value decomposition (SVD). Collaborative Filtering can be formulated by approximating a matrix X by using singular value decomposition. The winning team at the Netflix Prize competition used SVD matrix factorization models to produce product recommendations, for more information I recommend to read articles: Netflix Recommendations: Beyond the 5 stars and Netflix Prize and SVD. The general equation can be expressed as follows:

Given m x n matrix X :

U is an (m x r) orthogonal matrix

is an orthogonal matrix S is an (r x r) diagonal matrix with non-negative real numbers on the diagonal

is an diagonal matrix with non-negative real numbers on the diagonal V^T is an (r x n) orthogonal matrix

Elements on the diagnoal in S are known as singular values of X .

Matrix X can be factorized to U , S and V . The U matrix represents the feature vectors corresponding to the users in the hidden feature space and the V matrix represents the feature vectors corresponding to the items in the hidden feature space.

Now we can make a prediction by taking dot product of U , S and V^T .

Carelessly addressing only the relatively few known entries is highly prone to overfitting. SVD can be very slow and computationally expensive. More recent work minimizes the squared error by applying alternating least square or stochastic gradient descent and uses regularization terms to prevent overfitting. Alternating least square and stochastic gradient descent methods for CF will be covered in the future tutorials.

Review

We have covered how to implement simple Collaborative Filtering methods, both memory-based CF and model-based CF.

methods, both memory-based CF and model-based CF. Memory-based models are based on similarity between items or users, where we use cosine-similarity.

are based on similarity between items or users, where we use cosine-similarity. Model-based CF is based on matrix factorization where we use SVD to factorize the matrix.

is based on matrix factorization where we use SVD to factorize the matrix. Building recommender systems that perform well in cold-start scenarios (where little data is available on new users and items) remains a challenge. The standard collaborative filtering method performs poorly is such settings.

Looking for more ?

If you want to tackle your own recommendation system analysis, check out these data sets.

Note: The files are quite large in most cases, not all the links may stay up to host the data, but the majority of them still work. Or just Google for your own data set!

Movies Recommendation:

MovieLens — Movie Recommendation Data Sets http://www.grouplens.org/node/73

Yahoo! — Movie, Music, and Images Ratings Data Sets http://webscope.sandbox.yahoo.com/catalog.php?datatype=r

Jester — Movie Ratings Data Sets (Collaborative Filtering Dataset) http://www.ieor.berkeley.edu/~goldberg/jester-data/

Cornell University — Movie-review data for use in sentiment-analysis experiments http://www.cs.cornell.edu/people/pabo/movie-review-data/

Music Recommendation:

Last.fm — Music Recommendation Data Sets http://www.dtic.upf.edu/~ocelma/MusicRecommendationDataset/index.html

Yahoo! — Movie, Music, and Images Ratings Data Sets http://webscope.sandbox.yahoo.com/catalog.php?datatype=r

Audioscrobbler — Music Recommendation Data Sets http://www-etud.iro.umontreal.ca/~bergstrj/audioscrobbler_data.html

Amazon — Audio CD recommendations http://131.193.40.52/data/

Books Recommendation:

Institut für Informatik, Universität Freiburg — Book Ratings Data Sets http://www.informatik.uni-freiburg.de/~cziegler/BX/

Food Recommendation:

Chicago Entree — Food Ratings Data Sets http://archive.ics.uci.edu/ml/datasets/Entree+Chicago+Recommendation+Data

Healthcare Recommendation:

Nursing Home — Provider Ratings Data Set http://data.medicare.gov/dataset/Nursing-Home-Compare-Provider-Ratings/mufm-vy8d

Hospital Ratings — Survey of Patients Hospital Experiences http://data.medicare.gov/dataset/Survey-of-Patients-Hospital-Experiences-HCAHPS-/rj76-22dk

Dating Recommendation:

www.libimseti.cz — Dating website recommendation (collaborative filtering) http://www.occamslab.com/petricek/data/

Scholarly Paper Recommendation:

National University of Singapore — Scholarly Paper Recommendation http://www.comp.nus.edu.sg/~sugiyama/SchPaperRecData.html

Conclusion

It was a comprehensive articulation of statistics and algebra that typically goes behind perfecting a recommendation system on large scale. I sincerely hope, avid enthusiasts and practitioners would find it intriguing enough to reciprocate the pointers with caveats in mind. Don’t forget to share your interesting observations or empirical results in comment section while emulating the project. It really helps in expanding everyone’s knowledge horizon.