Recommender System — singular value decomposition (SVD) & truncated SVD | by Denise Chen | Towards Data Science

The most common method for recommendation systems often comes with Collaborating Filtering (CF) where it relies on the past user and item dataset. Two popular approaches of CF are latent factor models, which extract features from user and item matrices and neighborhood models, which finds similarities between products or users.
The neighborhood model is an item-oriented approach to discover the user preference based on the ratings given by the user for similar items. On the other hand, latent factor models such as Singular Value Decomposition (SVD) extract features and correlation from the user-item matrix. For example, when items are movies in different categories. SVD would generate factors when looking into the dimension space like action vs comedy, Hollywood vs Bollywood, or Marvel vs Disney. Mainly, we will focus on the latent factor model for the Singular Value Decomposition (SVD) approach.
In this article, you will learn the singular value decomposition and truncated SVD of the recommender system:
(1) Introduction to singular value decomposition
(2) Introduction to truncated SVD
(3) Hands-on experience of python code on matrix factorization
When it comes to dimensionality reduction, the Singular Value Decomposition (SVD) is a popular method in linear algebra for matrix factorization in machine learning. Such a method shrinks the space dimension from N-dimension to K-dimension (where K<N) and reduces the number of features. SVD constructs a matrix with the row of users and columns of items and the elements are given by the users’ ratings. Singular value decomposition decomposes a matrix into three other matrices and extracts the factors from the factorization of a high-level (user-item-rating) matrix.
Matrix U: singular matrix of (user*latent factors)Matrix S: diagonal matrix (shows the strength of each latent factor)Matrix U: singular matrix of (item*latent factors)
From matrix factorization, the latent factors show the characteristics of the items. Finally, the utility matrix A is produced with shape m*n. The final output of the matrix A reduces the dimension through latent factors’ extraction. From the matrix A, it shows the relationships between users and items by mapping the user and item into r-dimensional latent space. Vector X_i is considered each item and vector Y_u is regarded as each user. The rating is given by a user on an item as R_ui = X^T_i * Y_u. The loss can be minimized by the square error difference between the product of R_ui and the expected rating.
Regularization is used to avoid overfitting and generalize the dataset by adding the penalty.
Here, we add a bias term to reduce the error of actual versus predicted value by the model.
(u, i): user-item pairμ: the average rating of all itemsbi: average rating of item i minus μbu: the average rating given by user u minus μ
The equation below adds the bias term and the regularization term:
When it comes to matrix factorization technique, truncated Singular Value Decomposition (SVD) is a popular method to produce features that factors a matrix M into the three matrices U, Σ, and V. Another popular method is Principal Component Analysis (PCA). Truncated SVD shares similarity with PCA while SVD is produced from the data matrix and the factorization of PCA is generated from the covariance matrix. Unlike regular SVDs, truncated SVD produces a factorization where the number of columns can be specified for a number of truncation. For example, given an n x n matrix, truncated SVD generates the matrices with the specified number of columns, whereas SVD outputs n columns of matrices.
Truncated SVD can deal with sparse matrix to generate features’ matrices, whereas PCA would operate on the entire matrix for the output of the covariance matrix.
The metadata includes 45,000 movies listed in the Full MovieLens Dataset and movies are released before July 2017. Cast, crew, plot keywords, budget, revenue, posters, release dates, languages, production companies, countries, TMDB vote counts and vote averages are in the dataset. The scale of ratings is 1–5 and obtained from the official GroupLens website. The dataset is referred to from the Kaggle dataset.
Singular value decomposition (SVD) is a collaborative filtering method for movie recommendation. The aim for the code implementation is to provide users with movies’ recommendation from the latent features of item-user matrices. The code would show you how to use the SVD latent factor model for matrix factorization.
Random sample the rating dataset and generate the movie features with genres. Then, labelencode all the movies and users with respective unique ids.
num of users: 1105num of movies: 3000
Through each run of the epoch, the rmse is reduced and the final output reaches rmse 0.57. The number of batch size would affect the number of input data fed into the model for each run. Batch size, learning rate, and regularization term are tunable to optimize the model performance.
RMSE 2.1727233RMSE 2.101482RMSE 2.0310202RMSE 1.9610059RMSE 1.8911659RMSE 1.8213558RMSE 1.7515925RMSE 1.681992RMSE 1.612707RMSE 1.543902RMSE 1.4757496RMSE 1.408429RMSE 1.3421307RMSE 1.277059RMSE 1.2134355RMSE 1.1514966RMSE 1.0914934RMSE 1.0336862RMSE 0.9783424RMSE 0.9257237RMSE 0.87606686RMSE 0.82956517RMSE 0.7863303RMSE 0.7463626RMSE 0.7095342RMSE 0.67563176RMSE 0.6445249RMSE 0.6163493RMSE 0.5914116RMSE 0.5701855
The first 10 components of user x movie matrix s generated through truncated SVD. There are latent features in the reconstructed matrix showing a correlation with the user ratings for the rating prediction.
Since the genres column is in the list of the dictionary format, The column is preprocessed and extracted with several genres’ names separated by | format.
Take a 3000 random sample of users’ ratings from the dataset and create the pivot table with the index of Userid and columns of MovieID with the rating value. Then, the user matrix is generated with 2921x1739 users by the user matrix.
Take 3000 random samples of movies from the dataset and create the pivot table with the index of MovieID and columns of Userid with the rating value. Then, the movie matrix is generated with 3000x1105 users by the movie matrix.
From both user and rating matrix, 80% of data is used for training data and the rest 20% is for test data. For the train data, the reconstructed matrix is produced from 10 components of truncated SVD. The row*col length of matrices is movie_features.shape = (2400, 10) and user_features.shape = (2336, 10).
TSNE Visualization
TSNE transforms the high-dimensional space of data into a low-dimensional space of data and visualizes it. Perplexity is one of the tuneable features to take the balance of local and global data and suggest the number of close neighbors each point has.
Take the perplexity of 5 and 2 components of the movie features and the plot is produced and shows the clusters of movies. The correlated movies are clustered by the latent features produced from the TSNE method.
The label of the target data is the average users’ rating and round it to 1 decimal point. There are a total of 501 movies and 1108 users’ ratings. The size of the train and the target data are data.shape = (3000, 1105) and targets.shape = (3000,).
Train the model of GradientBoostingRegressor with the learning rate of 0.1 and 200 estimators. The loss function is calculated through mean squared error.
The final prediction is the average rating of each movie by all the ratings produced from the users. The final MSE is around 0.51, which is quite optimal for the average rating model.
       Iter       Train Loss   Remaining Time          1           0.3735            5.43s         2           0.3710            5.12s         3           0.3689            4.89s         4           0.3672            4.76s         5           0.3656            4.67s         6           0.3641            4.64s         7           0.3628            4.59s         8           0.3614            4.54s         9           0.3601            4.52s        10           0.3589            4.51s        20           0.3480            4.14s        30           0.3391            3.83s        40           0.3316            3.59s        50           0.3245            3.35s        60           0.3174            3.14s        70           0.3118            2.91s        80           0.3063            2.68s        90           0.3013            2.45s       100           0.2968            2.22s       200           0.2620            0.00sFinal MSE:0.5118555681581297
Singular value decomposition decomposes three matrices and the latent factors show the characteristics of the items. It reduces the dimension through latent factors’ extraction. By adding the regularization and bias term, it optimizes the model performance by minimizing the rmse error.Truncated SVD generates the matrices with the specified number of columns, whereas SVD outputs n columns of matrices. It decreases the number of output and better works on the sparse matrices for features output.
Singular value decomposition decomposes three matrices and the latent factors show the characteristics of the items. It reduces the dimension through latent factors’ extraction. By adding the regularization and bias term, it optimizes the model performance by minimizing the rmse error.
Truncated SVD generates the matrices with the specified number of columns, whereas SVD outputs n columns of matrices. It decreases the number of output and better works on the sparse matrices for features output.
Using truncated SVD to reduce dimensionalityhttps://subscription.packtpub.com/book/big_data_and_business_intelligence/9781783989485/1/ch01lvl1sec21/using-truncated-svd-to-reduce-dimensionalityRecommending movies using Truncated SVDhttps://github.com/saurabhmathur96/movie-recommendations/blob/master/notebooks/Recommending%20movies%20using%20Truncated%20SVD.ipynb
Using truncated SVD to reduce dimensionalityhttps://subscription.packtpub.com/book/big_data_and_business_intelligence/9781783989485/1/ch01lvl1sec21/using-truncated-svd-to-reduce-dimensionality
Recommending movies using Truncated SVDhttps://github.com/saurabhmathur96/movie-recommendations/blob/master/notebooks/Recommending%20movies%20using%20Truncated%20SVD.ipynb
3. Singular Value Decomposition (SVD) & Its Application In Recommender Systemhttps://analyticsindiamag.com/singular-value-decomposition-svd-application-recommender-system/#:~:text=In%20the%20context%20of%20the,given%20to%20items%20by%20users.
4. Recommending movies using SVD Matrix Factorisationhttps://github.com/saurabhmathur96/movie-recommendations/blob/master/notebooks/Recommending%20movies%20using%20SVD%20Matrix%20Factorisation.ipynb