Collaborative Filtering and Embeddings — Part 2 | by Shikhar Gupta | Towards Data Science

In Part 1, I’ve covered the basic idea behind collaborative filtering and the concept of embeddings and bias. It is recommended that you go through that post before going ahead.
Collaborative Filtering and Embeddings — Part 1
In this part I’ll talk about how we can implement collaborative filtering using a library called fastai developed by Jeremy Howard et al. This library is built on top of pytorch and is focused on easier implementation of machine learning and deep learning models.
Also, we’ll get to know how we can interpret and visualise embeddings using t-SNE, Plotly and Bokeh (Python interactive visualisation library that targets modern web browsers for presentation).
Most of the ideas presented in this post are derived from the Deep learning MOOC -v2 conducted by Jeremy Howard as part of the Data Institute. This post is just my attempt to share some amazing stuff that I’m learning during this course.
I have used Movielens [1] dataset (ml-latest-small). This dataset describes 5-star rating and free-text tagging activity from MovieLens, a movie recommendation service. It contains 100004 ratings and 1296 tag applications across 9125 movies. These data were created by 671 users between January 09, 1995 and October 16, 2016.
We’ll be using 2 files: ratings.csv and movies.csv
Before we get started we need 2 things:
A GPU enabled machine (local or AWS)
Install fastai library on your machine: pip install fastai
Note: At the end of the post I have explained in detail as to how to setup your system for fastai
Below is a step by step code walkthrough of the implementation using fastai. The underlying algorithm is the same as what we have discussed in Part 1.
We are looking at 2 files: ratings and movies
Ratings contain ratings by different users for different movies.
Movies contains metadata about movies. movieid is the key to join the 2 datasets.
#fastai functionval_idxs = get_cv_idxs(len(ratings)) #get validation indices 
We’ll divide our data into train and validation set. Our validation is 20% of our original dataset.
wd=2e-4 #weight decayn_factors = 50 #dimension of embedding vector
We’ll use weight decay to reduce overfitting. Also we have to define the dimension of our embedding vector.
#fastai functioncf = CollabFilterDataset.from_csv(path, 'ratings.csv', 'userId', 'movieId', 'rating') #creating a custom data loader
Now we have to create a data object for collaborative filtering. Think of it as something which will transform your raw data and prepare it in the form that is required by the model. from_csv implies that the input should be a csv file.
Parameters of the function:
path: path to the location of the csv file
ratings.csv : name of the csv file. It should be in the long format shown in figure 1
userID/movieID : column names of the 2 entities
rating : column name of the dependent variable that you want to predict
#create a learner (model) and specify the batch size and optimizer learn = cf.get_learner(n_factors, val_idxs, 64, opt_fn=optim.Adam) #fastai function
Next step is to create a model object which is a function of the data object that we have created. learner in fastai library is synonymous to model. This function takes the following parameters:
n_factors : Dimension of the embedding vector (50 in our case)
val_idxs : Row Indices from the ratings.csv file which have to be considered in validation
batch size : Number of rows that will be passed to the optimiser for each step of gradient descent. In our case 64 rows will be passed per iteration
opt_fn : Optimiser that we want to use. In our case we are using Adam. You have access to different optimisers in this library
#training with learning rate as 1e-2 learn.fit(1e-2, 2, wds=wd, cycle_len=1, cycle_mult=2, use_wd_sched=True) #fastai function
The final step of training is to actually train the model. Calling fit on the learner object trains the model and learn the right values in the embedding and bias matrix.
Parameters of the function:
learning rate : 1e-2 is the learning rate that we use for optimisation
wd : passing the weight decay
cycle_len/cycle_mult : These are fastai goodies that incorporate the state of the art methods for learning rate scheduling. The end of the post contains links to useful articles related to this.
use_wd_sched : Whether to use schedule for weight decay
When you run the above code, model will start training as shown below. You can observe training (left) and validation (right) loss after each epoch. Our loss function for optimisation is MSE (Mean squared error) .
preds = learn.predict() #prediction on validationmath.sqrt(metrics.mean_squared_error(y,preds)) #calculating RMSE
We’ll use the trained model to predict on validation and compute RMSE. We get an RMSE of ~.90 which is at par with the current benchmarks for this dataset.
y=learn.data.val_y #actual ratings for validationsns.jointplot(preds, y, kind='hex', stat_func=None);
We can also see that our predictions from the model are in tally with the actual ratings.
Now we’ll try to interpret the embeddings and bias and see if they capture some meaningful information.
We’ll focus on embeddings and bias for movies as we have actual titles for movies.
We know that our embedding vectors are of dimension 50. Visualising such high-dimensional vector is difficult and next to impossible.
t-distributed stochastic neighbour embedding (t-SNE) is a effective way to visualise them in lower dimensions while maintaining the spatial relationship between these vectors.
Using the above code I have reduced the dimensionality of embedding vectors to 2 (t-SNE components). Below we can visualise the 2 t-SNE components of embeddings for 3000 movies. Each point represents a movie.
You can play with this plot here: http://tsne.getforge.io/
Zooming in further we can see that movies from the same franchise are very close in the embedding space (almost overlapping). This shows that embeddings are not just some numbers optimised for reducing loss.
We expect that movies in the same franchise are more or less similar in characteristics like genre, director, cast among many other things. Closeness in embedding space reflects that embeddings are capturing these characteristics
Movie bias can be considered as a measure of actual goodness of the movie adjusting for the different rating patterns of different users.
Bias can be considered as a proxy for how good/popular the movie actually is
Let’s see whether this is reflected in the bias from our model
Looking at the list, I guess most of you’ll agree that this list seems more or less appropriate and closer to most of our expectations.
Ranking movies based on bias makes much more sense then just averaging the ratings given by all users. Opposed to the name, it’s actually making the rankings unbiased
For me this is the most interesting section. Below I have picked a user (userID:547) which has rated the maximum number of movies. Now I’m trying to see if there is any visible relationship between the ratings given by the user and the embedding vectors of different movies.
And guess what...
It’s quite evident that movies which are highly rated (black and red) are concentrated in one portion of the embedding space. Similarly, movies which are low rated (green and blue) are concentrated in another portion of the space.
Link to the html file of the plot. Download and open in browser: https://github.com/shik3519/collaborative-filtering/blob/master/t-SNE_cluster_rating.html
On similar lines I have tried to cluster movie embeddings based on the movie bias. The expectation is that movies with similar bias (measure of unbiased popularity as discussed) should be close in embedding space.
And we can see that in the below plot.
Movies which are good (or bad) have some characteristics in common which are getting captured by the embeddings
Link to the html file of the plot. Download and open in browser: https://github.com/shik3519/collaborative-filtering/blob/master/t-SNE_emb_cluster_bias.html
I was feeling fancy so I tried visualising 3 t-SNE components of movie embeddings clustered by bias. Each point represents a movie and the size and colour of the point is dependent on bias.
Here is the link to the actual plot: https://plot.ly/~shik1470/2/
Link to the data used for this plot: https://github.com/shik3519/collaborative-filtering/blob/master/t-sne1.csv
By now I guess you’ll be somewhat convinced that the idea of embeddings is quite powerful. This concept can be extended to any problem with structured data where you have a lot of categorical variables.
Each level of a categorical variable can be represented as a high dimensional vector which can capture relationships which label or one-hot encoding fails to capture. Label or one hot encoding assumes that every entity is independent of each other which is definitely not the case if you think about it.
Here is another cool post on usage of embeddings in structural settings that you should definitely checkout.
towardsdatascience.com
Learning rate selection and scheduling:
Learning rate selection and scheduling:
techburst.io
towardsdatascience.com
medium.com
2. t-SNE:
distill.pub
medium.com
www.kaggle.com
3. fastai:
Github repo of the library: https://github.com/fastai/fastai
fastai notebook on collaborative filtering
Instructions for setting up fastai:
4. GitHub repo for this post : This repo contains the notebook and plots shown in this post
[1] F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4, Article 19 (December 2015), 19 pages. DOI=http://dx.doi.org/10.1145/2827872