Serverless Recommendation System using PySpark and GCP
By: Badal Nabizade
Tags: Machine Learning, Cloud Computing, Towards Data Science, Apache Spark, Data Science

I will call the factor matrix for users as “X” and factor matrix for movies as “Y”. By the way, we don’t know what latent features represent and we don’t have to. — We just need to figure out the values of latent features, then determining unknown user ratings for each movie is just the matter of simple matrix multiplication.

To predict the values of latent features, we will use an alternating least squares approach.

A least squares approach in it’s basic forms means fitting some line to the data, measuring the sum of squared distances from all points to the line and trying to get an optimal fit for missing points.

With the alternating least squares approach the idea is the same, but we iteratively alternate between optimizing Y and fixing X and v.v. With this iterative optimization process main goal is to try to arrive closer to a factorized representation of our original data.

By doing so, we first estimate Y using X (we optimize the values of Y and keep X fixed) and estimate X by using Y (we optimize the values of X and keep Y fixed). After enough number of iterations (which is the hyper-parameter of model that we should determine), we are aiming to reach a convergence point where X and Y matrices are close enough to factorized representation of our original data and loss function for those matrices are minimized.

However, there is a minor issue about the data. Our data is sparse, meaning we don’t have full user-rating pairs for each movie :). This is the reason we are trying to predict those missing ratings and build a recommendation engine. So, we should penalize the movies that do not have ratings. After we do that, we don’t have to deal with the movies that don’t have ratings from the users and do not make any assumption around the movies that are not rated in the recommendation. So, to penalize non-rated movies, we’ll need a weight matrix where non-rated movies will have 0 as weight, and rated movies will have 1 as weight. I’ll call this wight matrix as “W”.

Since we have to optimize both X and Y matrices. There will be two cost functions corresponding to X and Y.

J(Xu) is a cost function for the user factor matrix and J(Yi) is a cost function for the movie factor matrix.

Mind the last terms in the above cost functions. Those are regularization terms and are the last hyper-parameter of this algorithm. They are there in order to avoid overfitting. In other words, they don’t let weights have high values during the optimization process. By the way, you should tune the lambda (regularization term) by using a cross-validation set (not test set) for the algorithm to generalize better on unseen data.

Optimization solutions for X and Y matrices are shown below.

Xu is a solution for the X matrix, Yi is a solution for the Y matrix. u is an u-th user in X matrix and i is an i-th movie in Y matrix.

In the above Optimization solutions, Wu represents weights of user latent features and Wi represents weights of movie latent features. qu and qi will be equal to respectively ru and ri when u th user has rated i-th movie. ru and ri represent the value of rating corresponding to the u-th user and value of rating corresponding to the i-th movie, respectively.

We can show the above algorithm in pseudocode as follows:

I implemented a naive version of this algorithm with Numpy in this notebook.