SVD: Where Model Tuning Goes Wrong | by Neil Chandarana | Towards Data Science

There are many ways to build recommender systems. Latent factor models are a family of techniques that seek a low dimension decomposition of a given ratings matrix. SVD is the name dubbed to a latent factor model involving parameter estimation by stochastic gradient descent made popular by Simon Funk in the context of making movie recommendations. This article demonstrates the process of building and testing an SVD recommender system on the Movie-Lens 100k dataset using the Surprise library in Python with a focus on hyperparameter tuning.
1 — Dataset Prerequisites
from surprise import Datasetdata = Dataset.load_builtin(‘ml-100k’)
Surprise is a scikit package for building and analysing recommender systems maintained by Nicolas Hug. Reading its documentation page, an objective of the package is to “alleviate the pain of dataset handling”. One way it does so is through built-in datasets. Movie-Lens 100k is one of those datasets and can be called with just the short command above.
from surprise import Dataset, Readerimport csvratingsPath = '[Your directory]/ratings.csv'reader = Reader(line_format='user item rating timestamp', sep=',', skip_lines=1)data = Dataset.load_from_file(ratingsPath, reader=reader)
Custom datasets can also be loaded from a file or pandas dataframe and in both cases, a Reader object must be defined before Surprise can parse the dataset. In my experience, the Reader object is solid as long the dataset contains a single rating per line in the format below where the order and separator are arbitrarily defined and timestamp is optional. In reality, the majority of datasets do not have this structure so some gutting may be required.
user:item:rating:timestamp
2 — Tuning Hyperparameters
from surprise import SVD,NormalPredictorfrom surprise.model_selection import GridSearchCVparam_grid = {'n_factors':[50,100,150],'n_epochs':[20,30],  'lr_all':[0.005,0.01],'reg_all':[0.02,0.1]}gs = GridSearchCV(SVD, param_grid, measures=['rmse'], cv=3)gs.fit(data)params = gs.best_params['rmse']svdtuned = SVD(n_factors=params['n_factors'], n_epochs=params['n_epochs'],lr_all=params['lr_all'], reg_all=params['reg_all')
With the dataset primed, the second step is to build the SVD algorithm. Surprise leverages scikit-learn’s GridSearchCV() and RandomisedSearchCV() methods and in just a few lines returns a set of tuned parameters bypassing the manual, trial-and-error process of adjusting hyperparameters.
We use the grid search method above. Once specifying hyperparameters and an array of potential values in the param_grid dictionary, GridSearchCV() calculates a score for each combination of hyperparameters on a k-fold cross validated dataset and returns the set of parameters that minimises the mean score across folds. Both the number of folds and the score can be selected by the user (hyper-hyperparameter?!) — we use 3 folds and RMSE accuracy score.
As with many machine learning algorithms, SVD has a lot of moving parts. We pick the number of factors, number of epochs, a single learning rate for all parameters and a single regularisation constant but the full list can be found in the library documentation here.
The final step is to use Surprise’s SVD() function, manually overriding the optimised hyperparameters.
The validity of SVD stems from a topic in Linear Algebra called Matrix decomposition and is modified to account for the sparse nature of ratings matrices. Derivation is left as an exercise for the reader but for those new to the methodology, Funk’s write-up here is a MUST read. It’s both entertaining and insightful!
3 — Evaluation Framework
from EvaluatorScript import Evaluatorfrom surprise import Dataset,SVD,NormalPredictorfrom surprise.model_selection import GridSearchCVdata = Dataset.load_builtin(‘ml-100k’)terminator = Evaluator(data)param_grid = {'n_factors': [50,100,150],'n_epochs': [20,30], 'lr_all': [0.005,0.01],'reg_all':[0.02,0.1]}gs = GridSearchCV(SVD, param_grid, measures=['rmse'], cv=3)gs.fit(data)params = gs.best_params['rmse']svdtuned = SVD(n_factors=params['n_factors'], n_epochs=params['n_epochs'],lr_all=params['lr_all'], reg_all=params['reg_all'])svd = SVD()random = NormalPredictor()terminator.addModel(svdtuned,'SVDtuned')terminator.addModel(svd,'SVD')terminator.addModel(random,'Random')terminator.evaluateModel()
The final step to the setup is to build an evaluation framework that looks like the above. This step is of personal preference. I like to build an Evaluator object that does the following,
a) Holds the dataset and all it’s splits (train/test, leave-one-out cross validated, etc).
b) Holds model objects via an .addModel() method.
c) Evaluates models via an .evaluateModel() method. In short this calls .fit() and .test() model object methods and evaluates predictions against a set of performance metrics using consistent dataset splits.
In the code excerpt above, I’ve created an instance of my Evaluator object that inherits from the EvaluationScript and I’ve named it terminator. After initialising terminator by feeding it our dataset, three model objects are built using Surprise and added. We’ve seen the tuned SVD model above but the remaining two models are an untuned SVD model and a random model.
I’ve found this framework to be SUPER versatile in that I can throw most model objects at it be they Surprise models or my own model classes. They simply require .fit() and .test() methods and a consistent prediction output format.
The Evaluator object and all its dependency scripts (including a Metric script) live in a Github folder (here). As a side note, I’d love to hear how you structure your machine learning scripts — I’m always open to hear alternate views on implementation!
With the help of the Surprise library in python, we have fitted a tuned SVD model, an untuned SVD model and a randomised model. While the tuned SVD model is our focus, we use the untuned SVD model and a set of randomly generated recommendations for relative comparison.
Tuned SVD hyperparameters:
{'n_factors': 150, 'n_epochs': 75, 'lr_all': 0.01, 'reg_all': 0.1}
Untuned SVD (default) hyperparameters:
{'n_factors': 100, 'n_epochs': 20, 'lr_all': 0.005, 'reg_all': 0.02}
Accuracy results:
          MAE      RMSE     Time (s)SVDtuned  0.6487   0.8469   199.3  SVD       0.6709   0.8719   134.4Random    1.1359   1.4228   109.4
The tuned model massively outperforms randomly generated recommendations on both MAE and RMSE and marginally outperforms the untuned model too. Great! Now, to see how this performance translates to actual movie recommendations we pick out a user in the dataset — User 56.
 — — — — — -User 56 Total Ratings: 46User 56 5 Star Ratings:Ace Ventura: When Nature Calls (1995)Seven (a.k.a. Se7en) (1995)Dumb & Dumber (Dumb and Dumber) (1994)Ace Ventura: Pet Detective (1994)Forrest Gump (1994)Lion King, The (1994)Jurassic Park (1993)Silence of the Lambs, The (1991) — — — — — -
Eyeballing the user’s 5-star rated movies, some obvious themes appear — comedy, crime & drama. This gives us a taste of the test user’s preference so let’s see how each model understands this user’s taste.
— — — — — -SVDtuned User's Top Recommendations:Belle Epoque (1992)Neon Genesis Evangelion: The End of Evangelion (Shin seiki Evangelion GekijÃ ́-ban: Air/Magokoro wo, kimi ni) (1997) Adam's Rib (1949) Dad's Army (1971)Three Billboards Outside Ebbing, Missouri (2017)Jetee, La (1962) Enter the Void (2009)Seve (2014) Cosmos Saving Face (2004)— — — — — -SVDUser's Top Recommendations:Shining, The (1980)Guess Who's Coming to Dinner (1967)Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981)Snatch (2000)On the Waterfront (1954)Good Will Hunting (1997)Matrix, The (1999)Kiss Kiss Bang Bang (2005)Hoop Dreams (1994)Sunset Blvd. (a.k.a. Sunset Boulevard) (1950)— — — — — -RandomUser's Top Recommendations:Saint, The (1997)Dish, The (2001)Bad Day at Black Rock (1955)Cry_Wolf (a.k.a. Cry Wolf) (2005)Osmosis Jones (2001)Neon Genesis Evangelion: The End of Evangelion (Shin seiki Evangelion GekijÃ ́-ban: Air/Magokoro wo, kimi ni) (1997)Sgt. Bilko (1996)O.J.: Made in America (2016)National Velvet (1944)Reckless Kelly (1994)-----------
Warning: Tuning gone wrong!
First thoughts on the tuned model’s recommendations — obscure. And not in a good way.
The tuned model’s top recommended movies appear way off the profile of our user. While some level of diversity is good (be it through international movies or otherwise), low familiarity to the user is often off-putting. In terms of understanding the user’s preferences, I cannot tell the difference between the tuned model and the random model. In fact, I do not recognise any top recommended movies...but that may say more about me that it does about the recommender system.
The untuned model’s top recommendations, however, largely fit the profile of user 56. The list appears to have done a pretty good job of identifying the user’s preference of drama as a genre whilst also blending crime and comedy. I’m impressed to see the wide range in release date of the recommended movies especially given the tight range in the user’s top list.
SVD, as a methodology, clearly has potential but quality of output remains very sensitive to hyperparameters. Our judgement so far has been for a single user in the dataset. We can analyse global performance with the use of some ‘beyond’ metrics.
          HitRate   5-StarHitRate  DiversitySVDtuned   0.0066      0.0163       0.7269  SVD        0.0295      0.0569       0.0319Random     0.0115      0.0081       0.0496
Somewhere between building a default model and what we thought was a tuned model, we’ve lost quality in our recommendations across the entire dataset. This is evident through the decline in HitRate — a sign that the model is struggling to identify user preferences from the items available.
Since HitRate is calculated on a leave-one-out cross validated dataset, 5-StarHitRate is simply the hit rate on the set of 5-star rated left out movies. A more granular hit rate metric if you will since we really only care about capturing the user’s preference. While both SVD models are better at identifying the top preferred movies, the additional 50 factors, 55 epochs, higher learning rate and higher penalty term have each culminated a WORSE ‘tuned’ model.
Why? The score we optimised in order to tune parameters in the grid search process did not align with the score we evaluated our models against.
The Surprise library lets us jump straight into a tuned model for our dataset potentially saving hours of manual work. It does, however, fall at a key hurdle. Failure to provide scores that really matter sadly renders the built-in tuning functionality useless for recommender systems. As we have seen above it is not enough to use stock accuracy metrics. In fact, they are the downfall of the tuned SVD model.
Model building is not an efficient truth-seeking process but it is an important step in the stairway to providing value to your users. In order to arrive at the best, usable model it’s important to be able spin up a working model quickly, tune hyperparameters effectively and analyse definitively. Regardless of the application, it is crucial to ensure the entire model process is aligned to the appropriate performance metrics.
Github: https://github.com/93tilinfinity/svd-recommender
Comments and feedback always welcome!