How to Build a Movie Recommendation System | by Ramya Vidiyala | Towards Data Science

Have you ever wondered how YouTube recommends content, or how Facebook recommends you, new friends? Perhaps you’ve noticed similar recommendations with LinkedIn connections, or how Amazon will recommend similar products while you’re browsing. All of these recommendations are made possible by the implementation of recommender systems.
Recommender systems encompass a class of techniques and algorithms that can suggest “relevant” items to users. They predict future behavior based on past data through a multitude of techniques including matrix factorization.
In this article, I’ll look at why we need recommender systems and the different types of users online. Then, I’ll show you how to build your own movie recommendation system using an open-source dataset.
Why Do We Need Recommender Systems?
Types of Recommender SystemsA) Content-Based Movie Recommendation SystemsB) Collaborative Filtering Movie Recommendation Systems
The Dataset
Designing a Movie Recommendation System
ImplementationStep 1: Matrix Factorization-based AlgorithmStep 2: Creating Handcrafted FeaturesStep 3: Creating a Final Model for our Movie Recommendation System
Performance Metrics
Summary
We now live in what some call the “era of abundance”. For any given product, there are sometimes thousands of options to choose from. Think of the examples above: streaming videos, social networking, online shopping; the list goes on. Recommender systems help to personalize a platform and help the user find something they like.
The easiest and simplest way to do this is to recommend the most popular items. However, to really enhance the user experience through personalized recommendations, we need dedicated recommender systems.
From a business standpoint, the more relevant products a user finds on the platform, the higher their engagement. This often results in increased revenue for the platform itself. Various sources say that as much as 35–40% of tech giants’ revenue comes from recommendations alone.
Now that we understand the importance of recommender systems, let’s have a look at types of recommendation systems, then build our own with open-sourced data!
Machine learning algorithms in recommender systems typically fit into two categories: content-based systems and collaborative filtering systems. Modern recommender systems combine both approaches.
Let’s have a look at how they work using movie recommendation systems as a base.
Content-based methods are based on the similarity of movie attributes. Using this type of recommender system, if a user watches one movie, similar movies are recommended. For example, if a user watches a comedy movie starring Adam Sandler, the system will recommend them movies in the same genre or starring the same actor, or both. With this in mind, the input for building a content-based recommender system is movie attributes.
With collaborative filtering, the system is based on past interactions between users and movies. With this in mind, the input for a collaborative filtering system is made up of past data of user interactions with the movies they watch.
For example, if user A watches M1, M2, and M3, and user B watches M1, M3, M4, we recommend M1 and M3 to a similar user C. You can see how this looks in the figure below for clearer reference.
This data is stored in a matrix called the user-movie interactions matrix, where the rows are the users and the columns are the movies.
Now, let’s implement our own movie recommendation system using the concepts discussed above.
For our own system, we’ll use the open-source MovieLens dataset from GroupLens. This dataset contains 100K data points of various movies and users.
We will use three columns from the data:
userId
movieId
rating
You can see a snapshot of the data in figure 3, below:
To obtain recommendations for our users, we will predict their ratings for movies they haven’t watched yet. Movies are then indexed and suggested to users based on these predicted ratings.
To do this, we will use past records of movies and user ratings to predict their future ratings. At this point, it’s worth mentioning that in the real world, we will likely encounter new users or movies without a history. Such situations are called cold start problems.
Let’s take a brief look at how cold start problems can be addressed.
Cold start problems can be handled by recommendations based on meta-information, such as:
For new users, we can use their location, age, gender, browser, and user device to predict recommendations.
For new movies, we can use genre, cast, and crew to recommend it to target users.
For our recommender system, we’ll use both of the techniques mentioned above: content-based and collaborative filtering. To find the similarity between movies for our content based method, we’ll use a cosine similarity function. For our collaborative filtering method, we’ll use a matrix factorization technique.
The first step towards this is creating a matrix factorization based model. We’ll use the output of this model and a few handcrafted features to provide inputs to the final model. The basic process will look like this:
Step 1: Build a matrix factorization-based model
Step 2: Create handcrafted features
Step 3: Implement the final model
We’ll look at these steps in greater detail below.
Matrix factorization is a class of collaborative filtering algorithms used in recommender systems. This family of methods became widely known during the Netflix prize challenge due to how effective it was.
Matrix factorization algorithms work by decomposing the user-movie interaction matrix into the product of two lower dimensionality rectangular matrices, say U and M. The decomposition is done in such a way that the product results in almost similar values to the user-movie interaction matrix. Here, U represents the user matrix, M represents the movie matrix, n is the number of users, and m is the number of movies.
Each row of the user matrix represents a user and each column of the movie matrix represents a movie.
Once we obtain the U and M matrices, based on the non-empty cells in the user-movie interaction matrix, we perform the product of U and M and predict the values of non-empty cells in the user-movie interaction matrix.
To implement matrix factorization, we use a simple Python library named Surprise, which is for building and testing recommender systems. The data frame is converted into a train set, a format of data set to be accepted by the Surprise library.
from surprise import SVDimport numpy as npimport surprisefrom surprise import Reader, Dataset# It is to specify how to read the data frame.reader = Reader(rating_scale=(1,5))# create the traindata from the data frametrain_data_mf = Dataset.load_from_df(train_data[['userId', 'movieId', 'rating']], reader)# build the train set from traindata. #It is of dataset format from surprise librarytrainset = train_data_mf.build_full_trainset()svd = SVD(n_factors=100, biased=True, random_state=15, verbose=True)svd.fit(trainset)
Now the model is ready. We’ll store these predictions to pass to the final model as an additional feature. This will help us incorporate collaborative filtering into our system.
#getting predictions of train settrain_preds = svd.test(trainset.build_testset())train_pred_mf = np.array([pred.est for pred in train_preds])
Note that we have to perform the above steps for test data also.
Let’s convert the data in the data frame format into a user-movie interaction matrix. Matrices used in this type of problem are generally sparse because there’s a high chance users may only rate a few movies.
The advantages of the sparse matrix format of data, also called CSR format, are as follows:
efficient arithmetic operations: CSR + CSR, CSR * CSR, etc.
efficient row slicing
fast matrix-vector products
scipy.sparse.csr_matrix is a utility function that efficiently converts the data frame into a sparse matrix.
# Creating a sparse matrixtrain_sparse_matrix = sparse.csr_matrix((train_data.rating.values, (train_data.userId.values, train_data.movieId.values)))
‘train_sparse_matrix’ is the sparse matrix representation of the train_data data frame.
We’ll create 3 sets of features using this sparse matrix:
Features which represent global averagesFeatures which represent the top five similar usersFeatures which represent the top five similar movies
Features which represent global averages
Features which represent the top five similar users
Features which represent the top five similar movies
Let’s take a look at how to prepare each in more detail.
The three global averages we’ll employ are:
The average ratings of all movies given by all usersThe average ratings of a particular movie given by all usersThe average ratings of all movies given by a particular user
The average ratings of all movies given by all users
The average ratings of a particular movie given by all users
The average ratings of all movies given by a particular user
train_averages = dict()# get the global average of ratings in our train set.train_global_average = train_sparse_matrix.sum()/train_sparse_matrix.count_nonzero()train_averages['global'] = train_global_averagetrain_averagesOutput: {‘global’: 3.5199769425298757}Next, let’s create a function which takes the sparse matrix as input and gives the average ratings of a movie given by all users, and the average rating of all movies given by a single user.# get the user averages in dictionary (key: user_id/movie_id, value: avg rating)def get_average_ratings(sparse_matrix, of_users):# average ratings of user/axesax = 1 if of_users else 0 # 1 - User axes,0 - Movie axes# ".A1" is for converting Column_Matrix to 1-D numpy arraysum_of_ratings = sparse_matrix.sum(axis=ax).A1# Boolean matrix of ratings ( whether a user rated that movie or not)is_rated = sparse_matrix!=0# no of ratings that each user OR movie..no_of_ratings = is_rated.sum(axis=ax).A1# max_user and max_movie ids in sparse matrixu,m = sparse_matrix.shape# create a dictionary of users and their average ratings..average_ratings = { i : sum_of_ratings[i]/no_of_ratings[i]for i in range(u if of_users else m)if no_of_ratings[i] !=0}#return that dictionary of average ratingsreturn average_ratings
The average rating is given by a user:
train_averages['user'] = get_average_ratings(train_sparse_matrix, of_users=True)
Average ratings are given for a movie:
train_averages['movie'] = get_average_ratings(train_sparse_matrix, of_users=False)
In this set of features, we will create the top 5 similar users who rated a particular movie. The similarity is calculated using the cosine similarity between the users.
# compute the similar Users of the "user"user_sim = cosine_similarity(train_sparse_matrix[user], train_sparse_matrix).ravel()top_sim_users = user_sim.argsort()[::-1][1:] # we are ignoring 'The User' from its similar users.# get the ratings of most similar users for this movietop_ratings = train_sparse_matrix[top_sim_users, movie].toarray().ravel()# we will make it's length "5" by adding movie averages totop_sim_users_ratings = list(top_ratings[top_ratings != 0][:5])top_sim_users_ratings.extend([train_averages['movie'][movie]]*(5 -len(top_sim_users_ratings)))
In this set of features, we obtain the top 5 similar movies rated by a particular user. This similarity is calculated using the cosine similarity between the movies.
# compute the similar movies of the "movie"movie_sim = cosine_similarity(train_sparse_matrix[:,movie].T,train_sparse_matrix.T).ravel()top_sim_movies = movie_sim.argsort()[::-1][1:]# we are ignoring 'The User' from its similar users.# get the ratings of most similar movie rated by this usertop_ratings = train_sparse_matrix[user, top_sim_movies].toarray().ravel()# we will make it's length "5" by adding user averages totop_sim_movies_ratings = list(top_ratings[top_ratings != 0][:5])top_sim_movies_ratings.extend([train_averages['user'][user]]*(5-len(top_sim_movies_ratings)))
We append all these features for each movie-user pair and create a data frame. Figure 5 is a snapshot of our data frame.
Here’s a more detailed breakdown of its contents:
GAvg: Average rating of all the ratings
Similar users rating of this movie: sur1, sur2, sur3, sur4, sur5 ( top 5 similar users who rated that movie )
Similar movies rated by this user: smr1, smr2, smr3, smr4, smr5 ( top 5 similar movies rated by user)
UAvg: User AVerage rating
MAvg: Average rating of this movie
rating: Rating of this movie by this user.
Once we have these 13 features ready, we’ll add the Matrix Factorization output as the 14th feature. In Figure 6 you can see a snapshot of our data after adding the output from Step 1.
The last column, named, mf_svd, is the additional column that contains the output of the model performed in Step 1.
To create our final model, let’s use XGBoost, an optimized distributed gradient boosting library.
# prepare train datax_train = final_data.drop(['user', 'movie','rating'], axis=1)y_train = final_data['rating']# initialize XGBoost modelxgb_model = xgb.XGBRegressor(silent=False, n_jobs=13,random_state=15,n_estimators=100)# fit the modelxgb_model.fit(x_train, y_train, eval_metric = 'rmse')
There are two main ways to evaluate a recommender system’s performance: Root Mean Squared Error (RMSE) and Mean Absolute Percentage Error (MAPE). RMSE measures the squared loss, while MAPE measures the absolute loss. Lower values mean lower error rates and thus better performance.
Both are good as they allow for easy interpretation. Let’s take a look at what each of them is:
RMSE is the square root of the average of squared errors and is given by the below formula.
Where:r is the actual rating,r^ is the predicted ratings andN is the total number of predictions
MAPE measures the error in percentage terms. It is given by the formula below:
Where:r is the actual rating,r^ is the predicted ratings andN is the total number of predictions
#dictionaries for storing train and test resultstest_results = dict()# from the trained model, get the predictionsy_est_pred = xgb_model.predict(x_test)# get the rmse and mape of train datarmse = np.sqrt(np.mean([ (y_test.values[i] - y_test_pred[i])**2 for i inrange(len(y_test_pred)) ]))mape = np.mean(np.abs( (y_test.values- y_test_pred)/y_true.values )) * 100# store the results in train_results dictionarytest_results = {'rmse': rmse_test, 'mape' : mape_test, 'predictions' : y_test_pred}
Our model resulted in 0.67 RMSE, and 19.86 MAPE on the unseen test data, which is a good-to-go model. An RMSE value of less than 2 is considered good, and a MAPE less than 25 is excellent. That said, this model can be further enhanced by adding features that would be recommended based on the top picks dependent on location or genre. We could also test the efficacy of our various models in real-time through A/B testing.
In this article, we learned the importance of recommender systems, the types of recommender systems being implemented, and how to use matrix factorization to enhance a system. We then built a movie recommendation system that considers user-user similarity, movie-movie similarity, global averages, and matrix factorization. These concepts can be applied to any other user-item interactions systems.
Thanks for reading! If you would like to experiment with this custom dataset yourself, you can download the annotated data on GroupLens and see my code at Github.
Thanks for the read. This article is originally published here. I am going to write more beginner-friendly posts in the future too. Follow me up on Medium to be informed about them. I welcome feedback and can be reached out on Twitter ramya_vidiyala and LinkedIn RamyaVidiyala. Happy learning!