Machine Learning for Building Recommender System in Python | by Yuefeng Zhang, PhD | Towards Data Science

Recommender systems are widely used in product recommendations such as recommendations of music, movies, books, news, research articles, restaurants, etc. [1][5][9][10].
There are two popular methods for building recommender systems:
collaborative filtering [3][4][5][10]
Content-based filtering [6][9]
The collaborative filtering method [5][10] predicts (filters) the interests of a user on a product by collecting preferences information from many other users (collaborating). The assumption behind the collaborative filtering method is that if a person P1 has the same opinion as another person P2 on an issue, P1 is more likely to share P2’s opinion on a different issue than that of a randomly chosen person [5].
Content-based filtering method [6][9] utilizes product features/attributes to recommend other products similar to what the user likes, based on other users’ previous actions or explicit feedback such as rating on products.
A recommender system may use either or both of these two methods.
In this article, I use the Kaggle Netflix prize data [2] to demonstrate how to use model-based collaborative filtering method to build a recommender system in Python.
The rest of the article is arranged as follows:
Overview of collaborative filtering
Build recommender system in Python
Summary
As described in [5], the main idea behind collaborative filtering is that one person often gets the best recommendations from another with similar interests. Collaborative filtering uses various techniques to match people with similar interests and make recommendations based on shared interests.
The high-level workflow of a collaborative filtering system can be described as follows:
A user rates items (e.g., movies, books) to express his or her preferences on the items
The system treats the ratings as an approximate representation of the user’s interest in items
The system matches this user’s ratings with other users’ ratings and finds the people with the most similar ratings
The system recommends items that the similar users have rated highly but not yet being rated by this user
Typically a collaborative filtering system recommends products to a given user in two steps [5]:
Step 1: Look for people who share the same rating patterns with the given user
Step 2: Use the ratings from the people found in step 1 to calculate a prediction of a rating by the given user on a product
This is called user-based collaborative filtering. One specific implementation of this method is the user-based Nearest Neighbor algorithm.
As an alternative, item-based collaborative filtering (e.g., users who are interested in x also interested in y) works in an item-centric manner:
Step 1: Build an item-item matrix of the rating relationships between pairs of items
Step 2: Predict the rating of the current user on a product by examining the matrix and matching that user’s rating data
There are two types of collaborative filtering system:
Model-based
Memory-based
In a model-based system, we develop models using different machine learning algorithms to predict users’ rating of unrated items [5]. There are many model-based collaborative filtering algorithms such as Matrix factorization algorithms (e.g., singular value decomposition (SVD), Alternating Least Squares (ALS) algorithm [8]), Bayesian networks, clustering models, etc.[5].
A memory-based system uses users’ rating data to compute the similarity between users or items. Typical examples of this type of systems are neighbourhood-based method and item-based/user-based top-N recommendations [5].
This article describes how to build a model-based collaborative filtering system using the SVD model.
This section describes how to build a recommender system in Python.
There are multiple Python libraries available (e.g., Python scikit Surprise [7], Spark RDD-based API for collaborative filtering [8]) for building recommender systems. I use the Python scikit Surprise library in this article for demonstration purpose.
The Surprise library can be installed as follows:
pip install scikit-surprise
As described before, I use the Kaggle Netflix prize data [2] in this article. There are multiple data files for different purposes. The following data files are used in this article:
training data:
combined_data_1.txt
combined_data_2.txt
combined_data_3.txt
combined_data_4.txt
Movie titles data file:
movie_titles.csv
The training dataset is too big to be handled on a Laptop. Thus I only load the first 100,000 records from each of the training data files for demonstration purpose.
Once training data files have been downloaded onto a local machine, the first 100,000 records from each of the training data files can be loaded into memory as Pandas DataFrames as follows:
def readFile(file_path, rows=100000):    data_dict = {'Cust_Id' : [], 'Movie_Id' : [], 'Rating' : [], 'Date' : []}    f = open(file_path, "r")    count = 0    for line in f:        count += 1        if count > rows:            break                    if ':' in line:            movidId = line[:-2] # remove the last character ':'            movieId = int(movidId)        else:            customerID, rating, date = line.split(',')            data_dict['Cust_Id'].append(customerID)            data_dict['Movie_Id'].append(movieId)            data_dict['Rating'].append(rating)            data_dict['Date'].append(date.rstrip("\n"))    f.close()                return pd.DataFrame(data_dict)df1 = readFile('./data/netflix/combined_data_1.txt', rows=100000)df2 = readFile('./data/netflix/combined_data_2.txt', rows=100000)df3 = readFile('./data/netflix/combined_data_3.txt', rows=100000)df4 = readFile('./data/netflix/combined_data_4.txt', rows=100000)df1['Rating'] = df1['Rating'].astype(float)df2['Rating'] = df2['Rating'].astype(float)df3['Rating'] = df3['Rating'].astype(float)df4['Rating'] = df4['Rating'].astype(float)
The resulting different DataFrames for different portions of training data are combined into one as follows:
df = df1.copy()df = df.append(df2)df = df.append(df3)df = df.append(df4)df.index = np.arange(0,len(df))df.head(10)
The movie titles file can be loaded into memory as Pandas DataFrame:
df_title = pd.read_csv('./data/netflix/movie_titles.csv', encoding = "ISO-8859-1", header = None, names = ['Movie_Id', 'Year', 'Name'])df_title.head(10)
The Dataset module in Surprise provides different methods for loading data from files, Pandas DataFrames, or built-in datasets such as ml-100k (MovieLens 100k) [4]:
Dataset.load_builtin()
Dataset.load_from_file()
Dataset.load_from_df()
I use the load_from_df() method to load data from Pandas DataFrame in this article.
The Reader class in Surprise is to parse a file containing users, items, and users’ ratings on items. The default format is that each rating is stored in a separate line in the following order separated by space: user item rating
This order and the separator are configurable using the following parameters:
line_format is a string like “item user rating” to indicate the order of the data with field names separated by a space
sep is used to specify separator between fields, such as space, ‘,’, etc.
rating_scale is to specify the rating scale. The default value is (1, 5)
skip_lines is to indicate the number of lines to skip at the beginning of the file and the default is 0
I use the default settings in this article. The item, user, rating correspond to the columns of Cust_Id, Movie_Id, and Rating of the DataFrame respectively.
The Surprise library [7] contains the implementation of multiple models/algorithms for building recommender systems such as SVD, Probabilistic Matrix Factorization (PMF), Non-negative Matrix Factorization (NMF), etc. The SVD model is used in this article.
The following code is to load data from Pandas DataFrame and create a SVD model instance:
from surprise import Reader, Dataset, SVDfrom surprise.model_selection.validation import cross_validatereader = Reader()data = Dataset.load_from_df(df[['Cust_Id', 'Movie_Id', 'Rating']], reader)svd = SVD()
Once the data and model for product recommendation are ready, the model can be evaluated using cross-validation as follows:
# Run 5-fold cross-validation and print resultscross_validate(svd, data, measures=['RMSE', 'MAE'], cv=5, verbose=True)
The following are the results of the cross validation of the SVD model:
Once the model has been evaluated to our satisfaction, then we can re-train the model using the entire training dataset:
trainset = data.build_full_trainset()svd.fit(trainset)
After a recommendation model has been trained appropriately, it can be used for prediction.
For example, given a user (e.g., Customer Id 785314), we can use the trained model to predict the ratings given by the user on different products (i.e., Movie titles):
titles = df_title.copy()titles['Estimate_Score'] = titles['Movie_Id'].apply(lambda x: svd.predict(785314, x).est)
To recommend products (i.e., movies) to the given user, we can sort the list of movies in decreasing order of predicted ratings and take the top N movies as recommendations:
titles = titles.sort_values(by=['Estimate_Score'], ascending=False)titles.head(10)
The following are the top 10 movies to be recommended to the user with Customer Id 785314:
In this article, I used the scikit Surprise library [7] and the Kaggle Netflix prize data [2] to demonstrate how to use model-based collaborative filtering method to build a recommender system in Python.
As described at the beginning of this article, the dataset is too big to be handled on a laptop or any typical single personal computer. Thus I only loaded the first 100,000 records from each of the training dataset files for demonstration purpose.
In the settings of real applications, I would recommend to use Surprise with Koalas or use the ALS algorithm in Spark MLLib to implement collaborative filtering system and run it on Spark cluster [8].
The Jupyter notebook with all of the source code used in this article is available in Github [11].
DLao, Netflix — Movie recommendationNetflix Prize data, Dataset from Netflix’s competition to improve their reccommendation algorithmRecommender systemA. Ajitsaria, Build a Recommendation Engine With Collaborative FilteringCollaborative filteringContent-based FilteringSurpriseS. Ryza, U. Laserson, et. al., Advanced Analytics with Spark, O’Reilly, April 2015N.S. Chauhan, How to build a Restaurant Recommendation Engine (part-1)N.S. Chauhan, How to build a Restaurant Recommendation Engine (part-2)Y. Zhang, Jupyter notebook in Github
DLao, Netflix — Movie recommendation
Netflix Prize data, Dataset from Netflix’s competition to improve their reccommendation algorithm
Recommender system
A. Ajitsaria, Build a Recommendation Engine With Collaborative Filtering
Collaborative filtering
Content-based Filtering
Surprise
S. Ryza, U. Laserson, et. al., Advanced Analytics with Spark, O’Reilly, April 2015
N.S. Chauhan, How to build a Restaurant Recommendation Engine (part-1)
N.S. Chauhan, How to build a Restaurant Recommendation Engine (part-2)
Y. Zhang, Jupyter notebook in Github