Movie Recommender System: Part 1. Learn how to build a Recommender system... | by Diven Sambhwani | Towards Data Science

All entertainment websites or online stores have millions/billions of items. It becomes challenging for the customer to select the right one. At this place, recommender systems come into the picture and help the user to find the right item by minimizing the options.
What are recommender systems?
It helps the user to select the right item by suggesting a presumable list of items and so it has become an integral part of e-commerce, movie and music rendering sites and the list goes on. They are becoming one of the most popular applications of machine learning which has gained importance in recent years. The two most popular ways it can be approached/built are:
1- Content-based Filtering
2- Collaborative Filtering
In this post, we will be focusing on the Matrix Factorization which is a method of Collaborative filtering.
Matrix Factorization
In collaborative filtering, matrix factorization is the state-of-the-art solution for sparse data problems, although it has become widely known since Netflix Prize Challenge.
“In the case of collaborative filtering, matrix factorization algorithms work by decomposing the user-item interaction matrix into the product of two lower dimensionality rectangular matrices. One matrix can be seen as the user matrix where rows represent users and columns are latent factors. The other matrix is the item matrix where rows are latent factors and columns represent items.”- Wikipedia
We will be working with MoiveLens Dataset, a movie rating dataset, to develop a recommendation system using the Surprise library “A Python scikit for recommender systems”. Let’s get started!
ratings = pd.read_csv('data/ratings.csv')ratings.head()
To load a data set from the above pandas data frame, we will use the load_from_df() method, we will also need a Reader object, and the rating_scale parameter must be specified.
The data frame must have three columns, corresponding to the user ids, the item ids, and the ratings in this order.
reader = Reader(rating_scale=(0.5, 5.0))data = Dataset.load_from_df(df[['userID', 'itemID', 'rating']], reader)
Rating Distribution
Rating Distribution by Item
Rating Distribution by User
Surprise is a Python scikit building and analyzing recommender systems that deal with explicit rating data. Maintained by Nicolas Hug.
With pip (you’ll need NumPy, and a C compiler. Windows users might prefer to use conda):
!pip install numpy!pip install scikit-surprise#For Windows usersconda install -c conda-forge scikit-surprise
We will use RMSE as our accuracy metric for the predictions.
We will be comparing SVD, NMF, Normal Predictor, KNN Basic and will be using the one which will have the least RMSE value.
Some understanding of the algorithms before we start applying.
1: Normal Predictor: It predicts a random rating based on the distribution of the training set, which is assumed to be normal. It’s a basic algorithm that does not do much work but that is still useful for comparing accuracies.
2: SVD: It got popularized by Simon Funk during the Netflix prize and is a Matrix Factorized algorithm. If baselines are not used, it is equivalent to PMF.
3: NMF: It is based on Non-negative matrix factorization and is similar to SVD.
4: KNN Basic: This is a basic collaborative filtering algorithm method.
benchmark = []# Iterate over all algorithmsfor algorithm in [SVD(), NMF(), NormalPredictor(), KNNBasic()]:# Perform cross validationresults = cross_validate(algorithm, data, measures=['RMSE'], cv=3, verbose=False)# Get results & append algorithm nametmp = pd.DataFrame.from_dict(results).mean(axis=0)tmp = tmp.append(pd.Series([str(algorithm).split(' ')[0].split('.')[-1]],index=['Algorithm']))benchmark.append(tmp)
As SVD has the least RMSE value we will tune the hyper-parameters of SVD.
Tuning algorithm parameters with GridSearchCV to find the best parameters for the algorithm.
Default values of SVD are :
n_factors — 100 | n_epochs — 20 | lr_all — 0.005 | reg_all — 0.02
param_grid = {'n_factors': [25, 30, 35, 40, 100], 'n_epochs': [15, 20, 25], 'lr_all': [0.001, 0.003, 0.005, 0.008],               'reg_all': [0.08, 0.1, 0.15, 0.02]}gs = GridSearchCV(SVD, param_grid, measures=['rmse', 'mae'], cv=3)gs.fit(data) algo = gs.best_estimator['rmse']print(gs.best_score['rmse']) print(gs.best_params['rmse'])#Assigning valuest = gs.best_params factors = t['rmse']['n_factors']epochs = t['rmse']['n_epochs'] lr_value = t['rmse']['lr_all']reg_value = t['rmse']['reg_all']
Output: 0.8682 {‘n_factors’: 35, ‘n_epochs’: 25, ‘lr_all’: 0.008, ‘reg_all’: 0.08}
Now as we have the right set of values for our hyper-parameters, Let’s split the data into train:test and fit the model.
trainset, testset = train_test_split(data, test_size=0.25)algo = SVD(n_factors=factors, n_epochs=epochs, lr_all=lr_value, reg_all=reg_value)predictions = algo.fit(trainset).test(testset)accuracy.rmse(predictions)
Output: RMSE: 0.8662
The following function will create a pandas data frame which will consist of these columns:
UID: user-id
iid: item id
Rui: the rating given by the user
est: rating estimated by the model
Iu: No of items rated by the user
UI: number of users that have rated this item
err: abs difference between predicted rating and the actual rating.
def get_Iu(uid):"""args: uid: the id of the userreturns:the number of items rated by the user"""try:    return len(trainset.ur[trainset.to_inner_uid(uid)])except ValueError: # user was not part of the trainset    return 0def get_Ui(iid):"""args:iid: the raw id of the itemreturns:the number of users that have rated the item."""try:   return len(trainset.ir[trainset.to_inner_iid(iid)])except ValueError:   return 0df_predictions = pd.DataFrame(predictions, columns=['uid', 'iid', 'rui', 'est', 'details'])df_predictions['Iu'] = df_predictions.uid.apply(get_Iu)df_predictions['Ui'] = df_predictions.iid.apply(get_Ui)df_predictions['err'] = abs(df_predictions.est - df_predictions.rui)
Best Predictions:
best_predictions = df_predictions.sort_values(by='err')[:10]
Worst predictions:
worst_predictions = df_predictions.sort_values(by='err')[-10:]
The worst predictions look pretty surprising. Let’s look in more details of item “3996”, rated 0.5, our SVD algorithm predicts 4.4
df.loc[df['itemID'] == 3996]['rating'].describe()temp = df.loc[df['itemID'] == 3996]['rating']# Create tracetrace = go.Histogram(x = temp.values, name = 'Ratings', xbins = dict(start = 0, end = 5, size=.3))# Create layoutlayout = go.Layout(title = 'Number of ratings item 3996 has received', xaxis = dict(title = 'Number of Ratings Per Item'), yaxis = dict(title = 'Count'), bargap = 0.2)# Create plotfig = go.Figure(data=[trace], layout=layout)iplot(fig)
It turns out, most of the ratings this Item received between “3 and 5”, only 1% of the users rated “0.5” and one “2.5” below 3. It seems that for each prediction, the users are some kind of outliers and the item has been rated very few times.
Looks like you’re enjoying the post!
You have just read Part 1 which covered how to build a model on explicit data using Surprise library.
Part 2 will cover how many movies we should recommend by calculating precision and recall at K and then recommending K movies.
For the complete code, you can find the Jupyter notebook here.
If you have any thoughts or suggestions please feel free to comment. You can also reach me through LinkedIn