Python: Implementing Matrix Factorization from Scratch!
By: Jacob Moore
Tags: Numpy, Recommendation System, Data Science, Python, Machine Learning

Python: Implementing Matrix Factorization from Scratch!

Credit: Pixabay

Have you ever wondered how Netflix is able to determine what content you’re likely to enjoy or how Amazon can bring products you’ll likely purchase to your attention? Or if you’re reading this — you’re likely very aware that these systems exist but aren’t sure how they function.

Content-Based Filtering vs Collaborative Filtering

Generally speaking, there are two main approaches to recommender systems: Content based and collaborative. There are, of course, hybrid approaches, and allegedly, a knowledge based system exists, too. See below: https://pdfs.semanticscholar.org/7e41/3d5661f185b4f19825da9220535cc04388ae.pdf

Without going down any unnecessary rabbit holes, content based means recommending content to you based on its similarities with other content without regarding the preferences of other users. Collaborative is the inverse, it does not concern itself with non-abstract features of an item, but rather uses the wisdom of the crowd to generate recommendations. An example of the former is Pandora’s recommendation system. They (painstakingly) crafted a musical genome of ~400 features. And songs are recommended to users by the degree to which they exhibit these features. (Think genre, song length, artist age, artist race, etc.) Collaborative filtering discovers patterns in user-item preferences and leverages these patterns to generate recommendations. For this article, we’ll focus only on collaborative filtering.

Within the collaborative filtering ecosystem, there are number of options available. For example, singular value decomposition (SVD) is one of the first such techniques, which (as the name suggests) decomposes the user-item preference matrix into three elements: the user-feature eigenvectors, the feature-item eigenvectors, and a diagonal matrix of eigenvalues. Decomposing a user-item matrix into terms of latent features is extremely useful. Users are understood in terms of the latent features that they prefer while items are understood in terms of the latent features and the degree to which they are present. In other words, distance metrics, such as cosine similarity can be used to compare users directly to items and generate the best match. Sure, you can compare users to users and items to items, but SVD is so much more powerful!

Another method, which we’ll focus on for the remainder of this article, is gradient descent based matrix factorization. The core idea here is to create parameters and iteratively update them to minimize some cost function. If this sounds like neural networks to you — you’re on the right track; we’re headed that direction! Note: Deep learning based methods do exist; however, we will not be using neurons or nonlinearities and we certainly will not be chaining multiple layers of neurons together (aka deep learning.)

Let’s start considering how we will approach this problem, beginning with understanding our parameters, matrix multiplication, and how we might iteratively update these parameters. Let’s consider the below:

The dot product of a user-feature row and feature-item column is a single user-item rating

Notice that each cell in both the user-feature matrix and the feature-item matrix is a parameter. (called weights in neural network lingo) We need to update these parameters iteratively through some cost function. But before we can dive into that, we need to know the user-item cells that any given user-feature cell and feature-item cell contribute to. For example, U1F1 and U1F2 contribute to U1I1 , U1I2 , and U1I3 . Matrix multiplication is executed as the dot product of the row from the left matrix with the column of the right matrix. As you’ll see, the row and column indices of the user-item matrix determine the whole row and column selected from the decomposed matrices.

user-feature cells and the user-item cells in which they contribute

Let’s consider another example. Notice that user-item cells that I’ve brought your attention to form a single column, not a row. This is because the cells F1I1 and F2I2 contribute to three different cells in the user-item matrix.

feature-item cells and the user-item cells in which they contribute

If it’s not obvious yet, we need to update our user-feature and feature-item parameters, but it’s important to note — each parameter will be updated by the average of three different gradients, each with respect to the given parameter.

The example below shows just one of the three gradients necessary to update U1F1 ; note that the other two gradients U1I2 and U1I3 , each (of course) with respect to U1F1 . When we update U1F1 we will average all three of these gradients. The intuition here is that if chased one gradient at a time, we might never converge on the optimal parameter values. By averaging the gradients, we can simultaneously take three steps towards the optimal decision.

MSE of U1I1 and gradient with respect to U1F1

Believe it or not — we now have all the building blocks we need to code up a matrix factorization solver in python! Let’s see the code. I’ve attached a Google Colab notebook below where you can see and run code to experiment with the results! We’ll dissect the code method by method in the class architecture shortly.

First, let’s create a class object and initialize its features. I’ve made the design decision of initializing the parameters to default values between 0.1 and 0.9. Additionally, we’ll define the MSE function as detailed earlier.

Second, let’s find just a single gradient. As detailed earlier this is one user-item cell with respect to one user-feature cell (or one feature-item cell).

But as we already discussed, it’s not enough to find just one gradient. We must consider the whole row (or whole column) when updating a given parameter.

Lastly, we need to define a method that trains the model. By default, we’ve set the learning rate to 0.1 and the iterations to 1000. This can be altered in the Google Colab referenced above. You’ll notice that the item feature values are increased in each iteration. Wasn’t the gradient negative? Yes! But in gradient descent, it’s standard to use the notation w -= learning_rate * gradient . In our case, the double negatives simply cancel out.

Now, let’s verify that our code works! As you can see, with only 2 latent features, there’s only so close we can get to the optimal solution. The error doesn’t reduce beyond ~3.54.

However, if we add just one more latent feature, we can very closely approach 0 error. In fact, when we multiply the matrices user-features and feature-items, we get the matrix we started with. Isn’t it beautiful!

I hope you’ve enjoyed today’s article. Please comment below if you’d like to see more content like this!