Build a Recommender System: Yelp Rating Prediction Example — Collaborative Filtering | by Zhongling Jiang | Towards Data Science

What’s in this article: I start this series of articles to introduce several popular recommender system techniques by explaining how they work and showing their application on dataset Yelp Challenge 2019. Each article will have a different focus, depending on whether complexity resides in data preprocessing, model concept or implementation. In this article, I will introduce one of most commonly used models called Collaborative Filtering (CF) and its implementation via a matrix factorization technique Singular Value Decomposition (SVD). This article will mainly focus on explaining concepts and accompany with key code walk-through. More details are included in code and comments posted on Github.
Note: This post is the result of a project done by a team of four (Bangwei Zhou, James Jungsuk Lee, Ujjwal Peshin and Zhongling Jiang). Great thanks to Brett Vintch for providing instructions and feedback on this project. Full codes will be available on this Github repository after each post. The code is written in Python.
Yelp Dataset Challenge [1] provides students a chance to perform research or analysis on Yelp’s data and share discoveries. Participants can use the data in innovative ways and find meaningful results to Yelp and its users.
The goal of this project is to compare different methodologies for recommending local business to users. This involves predicting rating values of business that users have not visited before based on their historical rating records. The performance of our models is mainly measured by Root Mean Square Error (RMSE). In addition, other types of metrics such as ranking metrics, coverage and novelty are also measured in order to evaluate the ranking quality, distribution of accuracy and diverseness of predictions. Also, model runtime is measured. They will be elaborated on in later articles.
The data comes in a zip file and can be accessed through this link. The file contains 6 JSON data files and our models mainly use the following:
reviews.json: contains 7 million reviews from 1.6 million users across 190 thousand business in 10 metropolitan areas. Each record is a rating and a full text review that user gives to a business on a certain date time.
business.json: contains business profile including location data, attributes, and categories.
user.json: contains user profile including the user’s friend mapping and all the metadata associated with the user e.g. reviews count, age of account, number of cool votes received, etc.
The original review dataset is large (> 5GB) so it is recommended to develop a smaller dataset that is representative of the entire dataset so as to speed up the development cycle. In addition, to provide sufficient per-user data for models to learn, or to avoid the cold-start problem, users who have fewer than 5 reviews are filtered out. Note that approaches that address cold-start will will be discussed in future articles but not be the main focus of this one.
To create a smaller dataset, downsampling is performed based on user ids. The reason we do not downsample on rating records is that this may create more users with fewer than 5 ratings, which interfere with the filtering step above. As a result, we subsample 20% of active users (with ≥5 reviews) and retain their associated ratings for modeling. The size of the resulting dataset also shrinks to ~20% compared with the original. Here is the data preparation process:
(i) Load Data — after downloading data zip file to a local repository, unzip review.json file and load json file into a pandas dataframe.
import pandas as pdimport tarfilefrom tqdm import tqdmimport jsonimport numpy as npimport timefrom copy import deepcopyzf = tarfile.open('yelp_dataset.tar') zf.extract('review.json')line_count = len(open("review.json").readlines())user_ids, business_ids, stars, dates, texts = [], [], [], [], []with open("review.json") as f:    for line in tqdm(f, total=line_count):        blob = json.loads(line)        user_ids += [blob["user_id"]]        business_ids += [blob["business_id"]]        stars += [blob["stars"]]        dates += [blob["date"]]        texts += [blob["text"]]ratings_ = pd.DataFrame(    {"user_id": user_ids, "business_id": business_ids, "rating": stars, "date": dates, "text": texts})
(ii) Retain only active users (≥ 5 ratings)
active_users = user_counts.loc[user_counts >= 5].index.tolist()ratings_ = ratings_.loc[ratings_.user_id.isin(active_users)]
(ii) Randomly sample 1/5 of users
SAMPLING_RATE = 1/5user_id_unique = ratings_.user_id.unique()user_id_sample = pd.DataFrame(user_id_unique, columns=['unique_user_id']) \                    .sample(frac= SAMPLING_RATE, replace=False, random_state=1)ratings_sample = ratings_.merge(user_id_sample, left_on='user_id', right_on='unique_user_id') \                    .drop(['unique_user_id'], axis=1)
(iii) Train- test- validation split — we evaluate model performance on the most recent rating given by each user. This way of creating testing set is considered an out-of-time approach, according to Tomas’s blog [6], and is different than normal in-time approach where train, validation and validation are generated randomly from the same time period. It is practically reasonable as we often use a user’s historical ratings to predict his future ratings. There may exist relationship in shift of data pattern that would otherwise gets dropped. Therefore, the latest rating of each user is held out as testing set. And then the second latest rating is held out as validation set.
# hold out last reviewratings_user_date = ratings_sample.loc[:, [‘user_id’, ‘date’]]ratings_user_date.date = pd.to_datetime(ratings_user_date.date)index_holdout = ratings_user_date.groupby([‘user_id’], sort=False)[‘date’].transform(max) == ratings_user_date[‘date’]ratings_holdout_ = ratings_sample[index_holdout]ratings_traincv_ = ratings_sample[~index_holdout]ratings_user_date = ratings_traincv_.loc[:, [‘user_id’, ‘date’]]index_holdout = ratings_user_date.groupby([‘user_id’], sort=False)[‘date’].transform(max) == ratings_user_date[‘date’]ratings_cv_ = ratings_traincv_[index_holdout]ratings_train_ = ratings_traincv_[~index_holdout]
After processing data, now let us look at models we are going to use.
Let us start by introducing a simple baseline model. An assumption this model makes is that each user has some natural bias towards ratings he gives, and hence rate businesses higher or lower in general. The model also assumes the businesses each have a higher or lower propensity to appeal to people in general. Also, the data has some underlying bias, and hence, each source of bias is included to model ratings using the following equation:
rui=μ + bu +bi
Here, μ is the average rating bias, and the parameters, bu and bi indicate the user and item bias respectively. For example, let us say that we want to find how a user ABC, might rate ‘SHAKE SHACK BURGERS’. Let us say that the average rating of all the restaurants in the data is 3.4, and in general, ‘SHAKE SHACK BURGERS’ is a highly rated restaurant, so it is rated 0.5 higher than the average. On the other hand, ‘ABC’ is a selective user and generally rates restaurants 0.4 lower than the average. So, the baseline estimate for the user ‘ABC’ for the restaurant ‘SHAKE SHACK BURGERS’ would be 3.4 + 0.5–0.4 = 3.5. The baseline function implemented estimates bu and bi to optimize following loss function (*):
∑ (rui − (μ + bu + bi))2 + λ (bu2 + bi2) where rui ∈ R_train, λ is regularization
A shortcut to implement bias model is to use Python surprise package, an easy-to-use library for recommender system. Readers may find this tutorial helpful when loading a pandas DataFrame as a special Dataset object and trying on a few basic models. In this example, BaselineOnly() with an input dictionary of hyperparameters specifies the biased baseline model object. The algo.fit() fits the model on training set whereas algo.test() yields predictions on validation set.
from surprise import SVDfrom surprise import accuracyfrom surprise import Readerfrom surprise import Datasetfrom surprise import BaselineOnlyreader = Reader(rating_scale = (0.0, 5.0))train_data = Dataset.load_from_df(trainset[['userID','itemID','rating']], reader)val_data = Dataset.load_from_df(valset[['userID','itemID','rating']], reader)test_data = Dataset.load_from_df(testset[['userID','itemID','rating']], reader)train_sr = train_data.build_full_trainset()val_sr_before = val_data.build_full_trainset()val_sr = val_sr_before.build_testset()test_sr_before = test_data.build_full_trainset()test_sr = test_sr_before.build_testset()bsl_options = {'method': 'als', 'n_epochs':3}bias_baseline = BaselineOnly(bsl_options)bias_baseline.fit(train_sr)predictions = bias_baseline.test(val_sr)
As one might notice, we specify {‘method’: ‘als’, ‘n_epochs’: 3} as hyperparameters. The ‘method’ refers to optimization method and ‘als’ is Alternating Least Square, a method that approaches solution for our baseline model’s loss function (*) via iterative updates on both bu and bi alternatively.
These hyperparameters can be tuned and then compared performance. A simple baseline model may bring us preliminary predictions of ratings, but just not sufficient. As bias are calculated across all users or all items, it cannot capture granularities among specific users or items. Therefore, we want to use a model that leverages these user-to-user or item-to-item underlying relationships more. In the next section, we will take a look at Collaborative Filtering.
Collaborative Filtering
Common problems in recommender systems involve two types of data: (i) user-item interactions, e.g. ratings, viewership (ii) user or item-specific attributes, e.g. textual profiles. Methods that use the former are referred to as collaborative filtering methods. They belong to a class of powerful techniques as they use ratings from multiple users in a collaborative way. The basic idea is that unspecified ratings can be predicted because observed ratings are often highly correlated across various users and items [2]. By identifying similar users, one could make inference about an unspecified rating by looking up ratings (for the same item) given by similar users.
There are two types of CF methods commonly used: neighborhood based methods and model based methods. Neighborhood based methods aim to identify similar users (or similar items) and make predictions using weighted average ratings from peer users (or items) group, e.g. using K-nearest neighbors. These methods are valued for simplicity in implementation as well as interpretation. However, their performance is limited especially when rating matrix is sparse, as it becomes difficult to identify sufficiently similar users who have rated the same item. On the other hand, model based method would have a summarized machine learning model, e.g. decision-tree, latent-factor model, etc., of the data upfront and treat rating prediction as a generalized regression or classification problem. For example, the approach that I introduce next belongs to latent factor model: matrix factorization.
Matrix Factorization and SVD
In this project, we use matrix factorization (MF), a class of model-based CF algorithm. Namely, MF decomposes a m (number of users) * n (number of items) user-item matrix into two matrices — m * k and k * n with a shared dimension k, as follow: R = X* Y. The space that has dimension k is called a latent space. But how is decomposition related to recommendation task? A more intuitive way to understand is that the goal of this approach is to map both users and items into a joint latent factor space of dimension k, such that user-item interactions are modeled as inner product in that space [3]. Let us take a look at the following graph:
Each users u is associated with a vector pu ∈ Rk and each items is associated with a vector qi ∈ Rk. Each dimension of vector represents that item or user’s affinity with the corresponding latent factor, or characteristic e.g. whether a restaurant being fine dining or casual restaurant, its price being more expensive or less, etc. Nicholas Hug’s blog provides really good interpretation of latent vectors [4]. Interestingly, it is likely to be observed that users who have similar tastes or items that have similar characteristics would lie geometrically close to each other in the latent space. Moreover, item i and user u that have a higher dot product i.e qit pu indicates higher rating rui that user u might offer, as the user and the item tend to share affinity over a similar set of latent factors.
Now the challenge is to identify such mapping (User u, pu) and (Item i, qi) since after this the rating rui can be approximated by their dot product, i.e rui ≃ qit pu. Notice how this form matches up with matrix decomposition R = X * Y. We can use singular value decomposition (SVD) to find this mapping:
R = P ∑ Qt
where P ∈ Rm x m, ∑ ∈ Rm x n, Q ∈ Rn x n, and P, Q are column-orthogonal matrices, ∑ is diagonal matrix. If we look at each row of P and each column of Qt separately, then we notice SVD allows us to model the rui ≃ qit pu relationship directly, as ∑ is simply a set of scalars.
In practice, SVD is implemented with regularization, namely regularized SVD [5], in order to avoid overfitting on modeling sparse rating matrix. Also, bias terms as in our baseline are also added so that the final approximation is as follow [3]:
rui=μ + bu +bi + qit pu
and loss function is
∑ (rui − (μ + bu + bi + qit pu))2 + λ ( |pu|2 + |qi|2 + bu2 + bi2)
This is exactly how surprise package implements SVD in the backend. With surprise SVD() function, we can build, tune and test model and in a few lines:
RMSE_tune = {}n_epochs = [5, 7, 10]  # the number of iteration of the SGD procedurelr_all = [0.002, 0.003, 0.005] # the learning rate for all parametersreg_all =  [0.4, 0.5, 0.6] # the regularization term for all parametersfor n in n_epochs:    for l in lr_all:        for r in reg_all:            algo = SVD(n_epochs = n, lr_all = l, reg_all = r)            algo.fit(train_sr)            predictions = algo.test(val_sr)            RMSE_tune[n,l,r] = accuracy.rmse(predictions)algo_real = SVD(n_epochs = 10, lr_all = 0.005, reg_all = 0.4)algo_real.fit(train_sr)predictions = algo_real.test(test_sr)
In this project, several evaluation metrics are used to measure model performance from different angles. They are:
Regression metrics, e.g. RMSE, MAE, R-squared
Ranking metrics, e.g. top-k recommendations, average ranking
Coverage
Novelty
Most directly, regression metrics tell us how close predicted ratings are to actual ratings. Although regression metrics are what we rely on to evaluate the quality of our predictions overall, they alone do not show us the full picture. From a slightly different perspective, we can view our problem as a ranking problem — that how likely we would recommend a business to a user which he visits next and then finds it good. In practice, only top-k businesses are recommended to users as a list and users tend to pay most attention to them, instead of the ones at the bottom of entire search result. Therefore, besides pursuing a low prediction error, we also care about whether top recommendations capture users’ taste as they are most relevant to user experience. It is worth noticing that a small difference in error metrics may lead to larger one in ranking metrics.
In addition, whether businesses we recommend are diversified enough to cover a good proportion of all existing businesses is a concern — measured by coverage metrics. Lastly, we want to measure if our models recommend novel options to users so that would avoid always showing same sets of businesses. We will take a closer look at these metrics in future articles. Definitions of these metrics are available in our code for reader’s reference too.
Comparing 2 model’s result on 20% training data:
# Bias BaselineRMSE: 1.3811MAE:  1.1598R^2:  0.1661Top_10: 0.1233Coverage:0.2044Serendipity: 0.9795Avg_rank: 526.6The algorithm takes 2.341 seconds.# Collaborative Filtering via SVD (CF-SVD)RMSE: 1.3595MAE: 1.1170R^2: 0.1920Top_10: 0.1140 Coverage: 0.4760Serendipity: 0.9672Avg_rank: 541.1The algorithm takes 83.351 seconds.
For regression metrics, CF-SVD shows better performance in terms of lower RMSE, MAE and higher R2. For ranking metrics, the result is mixed. CF-SVD recommends a wider range of businesses whereas baseline model provides top-10 recommendations that capture users’ taste slightly better.
A main downside of CF-SVD is that it takes a much longer run time, which may become a concern to scale. Readers are welcomed to further compare two models’ performance by training them on 100% data.
In this article, I have introduced Collaborative Filtering — a classical but powerful class of techniques for building recommender system and its one implementation through singular value decomposition using Python surprise library. The algorithm is powerful because it models user-item interactions by placing both users and items into a same latent space and exploiting their affinity with some structured, underlying, characteristic-capturing factors. From a matrix perspective, the algorithm captures both row interactions and column interactions during factorization. These are all reasons that bring CF’s performance to a decent level and make it popular baseline among recent research. Notice this is just a start, since we are utilizing rating data alone. Other information like user profile, business location, categories and time may also provide useful signals to accurately predict ratings. In the next article, I will introduce a few state-of-the-art models that comes into play when we want to integrate more information as mentioned above.
[1] Yelp Dataset Challenge (2019)
[2] C. C. Aggarwal, Recommender Systems The Textbook (2018), Cham: Springer International Publishing.
[3] Y. Koren, R. Bell, & C. Volinsky, Matrix Factorization Techniques for Recommender Systems (2009), Computer, 42(8), 30–37. doi: 10.1109/mc.2009.263
[4] N. Hug, Understanding matrix factorization for recommendation (part 2) — the model behind SVD (2017), Blog post.
[5] B. Webb, Netflix update: Try this at home (2006), The Evolution of Cybernetics.
[6] T. Dvorak, Why isn’t out-of-time validation more ubiquitous? (2019), Towards Data Science.