Prototyping a Recommender System Step by Step Part 2: Alternating Least Square (ALS) Matrix Factorization in Collaborative Filtering | by Kevin Liao | Towards Data Science

Part 1 of recommender systems can be found here
In the last post, we covered a lot of ground in how to build our own recommender systems and got our hand dirty with Pandas and Scikit-learn to implement a KNN item-based collaborative filtering movie recommender. The source code of the KNN recommender system can be found in my Github repo.In this post, we will talk about how to improve our movie recommender system with a more sophisticated machine learning technique: Matrix Factorization. Later in this post, we will discuss why we want to use matrix factorization in collaborative filtering and what is matrix factorization and how is it implemented in Spark.
During the last section in previous post, we asked our model for some movie recommendations. After we evaluated the list of recommended movies, we quickly identified two obvious limitations in our KNN approach. One is the “popularity bias”, the other is “item cold-start problem”. There will be another limitation, “scalability issue”, if the underlying training data is too big to fit in one machine
popularity bias: refers to system recommends the movies with the most interactions without any personalization
item cold-start problem: refers to when movies added to the catalogue have either none or very little interactions while recommender rely on the movie’s interactions to make recommendations
scalability issue: refers to lack of the ability to scale to much larger sets of data when more and more users and movies added into our database
All three above are very typical challenges for collaborative filtering recommender. They arrive naturally along with the user-movie (or movie-user) interaction matrix where each entry records an interaction of a user i and a movie j. In a real world setting, the vast majority of movies receive very few or even no ratings at all by users. We are looking at an extremely sparse matrix with more than 99% of entries are missing values.
With such a sparse matrix, what ML algorithms can be trained and reliable to make inference? To find solutions to the question, we are effectively solving a data sparsity problem.
In collaborative filtering, matrix factorization is the state-of-the-art solution for sparse data problem, although it has become widely known since Netflix Prize Challenge.
What is matrix factorization? Matrix factorization is simply a family of mathematical operations for matrices in linear algebra. To be specific, a matrix factorization is a factorization of a matrix into a product of matrices. In the case of collaborative filtering, matrix factorization algorithms work by decomposing the user-item interaction matrix into the product of two lower dimensionality rectangular matrices. One matrix can be seen as the user matrix where rows represent users and columns are latent factors. The other matrix is the item matrix where rows are latent factors and columns represent items.
How does matrix factorization solve our problems?
Model learns to factorize rating matrix into user and movie representations, which allows model to predict better personalized movie ratings for usersWith matrix factorization, less-known movies can have rich latent representations as much as popular movies have, which improves recommender’s ability to recommend less-known movies
Model learns to factorize rating matrix into user and movie representations, which allows model to predict better personalized movie ratings for users
With matrix factorization, less-known movies can have rich latent representations as much as popular movies have, which improves recommender’s ability to recommend less-known movies
In the sparse user-item interaction matrix, the predicted rating user u will give item i is computed as:
Rating of item i given by user u can be expressed as a dot product of the user latent vector and the item latent vector.
Notice in above formula, the number of latent factors can be tuned via cross-validation. Latent factors are the features in the lower dimension latent space projected from user-item interaction matrix. The idea behind matrix factorization is to use latent factors to represent user preferences or movie topics in a much lower dimension space. Matrix factorization is one of very effective dimension reduction techniques in machine learning.
Very much like the concept of components in PCA, the number of latent factors determines the amount of abstract information that we want to store in a lower dimension space. A matrix factorization with one latent factor is equivalent to a most popular or top popular recommender (e.g. recommends the items with the most interactions without any personalization). Increasing the number of latent factors will improve personalization, until the number of factors becomes too high, at which point the model starts to overfit. A common strategy to avoid overfitting is to add regularization terms to the objective function.
The objective of matrix factorization is to minimize the error between true rating and predicted rating:
Once we have an objective function, we just need a training routine (eg, gradient descent) to complete the implementation of a matrix factorization algorithm. This implementation is actually called Funk SVD. It is named after Simon Funk, who he shared his findings with the research community during Netflix prize challenge in 2006.
Although Funk SVD was very effective in matrix factorization with single machine during that time, it’s not scalable as the amount of data grows today. With terabytes or even petabytes of data, it’s impossible to load data with such size into a single machine. So we need a machine learning model (or framework) that can train on dataset spreading across from cluster of machines.
Alternating Least Square (ALS) is also a matrix factorization algorithm and it runs itself in a parallel fashion. ALS is implemented in Apache Spark ML and built for a larges-scale collaborative filtering problems. ALS is doing a pretty good job at solving scalability and sparseness of the Ratings data, and it’s simple and scales well to very large datasets.
Some high-level ideas behind ALS are:
Its objective function is slightly different than Funk SVD: ALS uses L2 regularization while Funk uses L1 regularization
Its training routine is different: ALS minimizes two loss functions alternatively; It first holds user matrix fixed and runs gradient descent with item matrix; then it holds item matrix fixed and runs gradient descent with user matrix
Its scalability: ALS runs its gradient descent in parallel across multiple partitions of the underlying training data from a cluster of machines
If you are interested in learning more about ALS, you can find more details in this paper: Large-scale Parallel Collaborative Filtering for the Netflix Prize
Just like other machine learning algorithms, ALS has its own set of hyper-parameters. We probably want to tune its hyper-parameters via hold-out validation or cross-validation.
Most important hyper-params in Alternating Least Square (ALS):
maxIter: the maximum number of iterations to run (defaults to 10)
rank: the number of latent factors in the model (defaults to 10)
regParam: the regularization parameter in ALS (defaults to 1.0)
Hyper-parameter tuning is a highly recurring task in many machine learning projects. We can code it up in a function to speed up the tuning iterations.
After tuning, we found the best choice of hyper-parameters: maxIter=10, regParam=0.05, rank=20
Now that we know we have a wonderful model for movie recommendation, the next question is: how do we take our wonderful model and productize it into a recommender system? Machine learning model productization is another big topic and I won’t get into details about it. In this post, I will show how to build a MVP (minimum viable product) version for ALS recommender.
To productize a model, we need to build a work flow around the model. Typical ML work flow roughly starts with data preparation via pre-defined set of ETL jobs, offline/online model training, then ingesting trained models to web services for production. In our case, we are going to build a very minimum version of movie recommender that just does the job. Our work flow is following:
A new user inputs his/her favorite movies, then system create new user-movie interaction samples for the modelSystem retrains ALS model on data with the new inputsSystem creates movie data for inference (in my case, I sample all movies from the data)System make rating predictions on all movies for that userSystem outputs top N movie recommendations for that user based on the ranking of movie rating predictions
A new user inputs his/her favorite movies, then system create new user-movie interaction samples for the model
System retrains ALS model on data with the new inputs
System creates movie data for inference (in my case, I sample all movies from the data)
System make rating predictions on all movies for that user
System outputs top N movie recommendations for that user based on the ranking of movie rating predictions
Here is a small snippet of the source code for our MVP recommender system:
This snippet demos our make_recommendations method in our recommender’s implementation. Please find the detailed source code for recommender application in my GitHub Repo.
Once we implemented the ALS recommender system in a python script as a small Pyspark program, we can submit our spark application to a cluster with Client Deploy Mode or Cluster Deploy Mode and enjoy the power of distributed computing.
Finally, we are done with the technical details and implementations. Now let’s ask our recommender for some movie recommendations. I will pretend a new user and input my favorite movie “Iron Man” again into this new recommender system. Let’s see what movies it recommends to me. Hopefully, they are not the same list of popular movies that I have watched many times before.
For demo purpose, I submit my spark application locally by running following commends in terminal: (instruction of commands can be found here)
spark-submit --master local[4] --driver-memory 4g              --executor-memory 8g src/als_recommender.py              --movie_name "Iron Man" --top_n 10
Yay!! Successfully run our movie recommender with Spark.
This new list of recommended movies are completely different than the list from previous KNN recommender, which is very interesting!! I have never watch any one of movies from this new list. I find it very surprising that the new recommender proposed me unusual movies. They might be too unusual for other users, which is problematic.
One idea to further improve our movie recommender system is to blend this new list of movie recommendations with the previous list from KNN recommender. We basically implement a hybrid recommender system and this hybrid recommender can offer both popular and less-know content to users.
In this post, we covered how to improve collaborative filtering recommender system with matrix factorization. We learned that matrix factorization can solve “popular bias” and “item cold-start” problems in collaborative filtering. We also leveraged Spark ML to implement distributed recommender system using Alternating Least Square (ALS). The Jupyter Notebook version for this blog post can be found here. If you want to play around my source code, you can find it here.
In my next post, we will dig deeper at Matrix Factorization techniques. We can develop a more generalized form of matrix factorization model with neural network implementation in Keras. Stay tuned! Until then, have fun with machine learning and recommenders!
Like what you read? Checkout more data science / machine learning projects at my Github: Kevin’s Data Science Portfolio