Recommender Systems from Learned Embeddings | by Nahid Alam | Towards Data Science

Recommender systems are fundamental building blocks of modern consumer web applications that seek to predict user preferences. There are different approaches to build a Recommender system. We are particularly interested in the Nearest Neighbor based approach. We look into the similarity between pairs of an item and recommend to the users
In this article, we will apply a Deep Neural Network on a movie dataset to recommend movies to users. The system will learn movie embeddings along with the model training process. We will use these embeddings to create the nearest neighbor database. Whenever a user clicks on a movie, the system will extract its embedding and find the nearest neighbor movies to recommend to the user.
If you find this article interesting, feel free to connect on LinkedIn.
The overall architecture of this system is shown in Figure 1
We will use Movie ID and User ID to generate their corresponding embeddings. These embeddings are generated through the model training process along with other parameters. Once we have the embeddings, we build a K-Nearest Neighbor (KNN) model. Then whenever there is a user, we can get that user’s embedding from our Neural Network model. We use this embedding to lookup in the KNN database and recommend top — K movies to this user.
One thing to note here is that this approach will not be able to handle unknown users. That is, if the User ID was absent during the model training process, we will not be able to recommend movies for that user. There are different ways to solve this problem. Such as finding similar users based on the demographic or other features and recommend based on that for the new user. But that discussion is out of scope for this article.
In this article, I will be using Keras (Tensorflow 2.0 as backend) to build our Deep Learning model.
For this post, we will be using MovieLens 100K dataset. The dataset contains 100K rows and 34 columns. The columns are shown below.
'user_id', 'movie_id', 'rating', 'unix_timestamp', 'title', 'release_date', 'video_release_date', 'imdb_url', 'genre_unknown','Action', 'Adventure', 'Animation', 'Children', 'Comedy', 'Crime','Documentary', 'Drama', 'Fantasy', 'Film-Noir', 'Horror', 'Musical','Mystery', 'Romance', 'Sci-Fi', 'Thriller', 'War', 'Western', 'year','genre', 'all_genres', 'age', 'sex', 'occupation', 'zip_code'
I won’t go into the details of Feature Engineering as it is out of scope for this post. The dataset has 943 unique users and 1682 unique movies.We will use user_id and movie_id as features and rating as the target of our embedding based model.
Embeddings are vector representations of an entity. You can represent any discrete entity to a continuous space through Embeddings [2]. Each item in the vector represents a feature or a combination of features for that entity.
For example, you can represent movies and user IDs as embeddings. For simplicity, let's assume that these embeddings are two-dimensional vector as shown in Figure -2. We will create movie and user embeddings and train them along with our neural network. The goal is to bring users and movies in a similar embedding space so we can apply the nearest neighbor based recommendation. Figure -2 shows a hypothetical example where the user_1 is closer to movie_1 and movie_2 in the embedding space than to movie_3. So we will recommend movie_1 and movie_2 to user_1.
With the above concept in mind, let's define our model. In this article, I am building a four-layer Deep Neural Network. You will define EMBEDDING_SIZE as the dimension of the vector you want for your embedding. Generally higher dimension is better because that captures more features although that might increase the training time. NUM_USERS is 943 and NUM_MOVIES is 1682 in this case. ROW_COUNT indicates the size of your training set.
As you see in the code above, we have created an embedding layer for movies (line 4) and user (line 11). It is worthwhile for a Machine Learning Engineer to understand how Keras Embedding layer works [3][4]. The output_dim indicated the length of the embedding vector which in our case is EMBEDDING_SIZE. The input_dim indicates the number of distinct entities — in this case either NUM_MOVIES or NUM_USERS. The input_length argument determines the size of each input sequence. Embedding layer initializes the embedding weights based on embedding_initializer parameter [3]. If you train these embedding layers along with the network, the output of these layers will get updated. This is what happens in our model.
I trained the model for 50 epochs with early_stopping set on validation loss unchanged in 10 consecutive epochs. You can play with these parameters to suit your needs.
We can plot the model to visualize its architecture
keras.utils.plot_model(model, show_shapes=True) 
We can plot the loss curve over different epochs
Figure 4 shows that the training loss decreases over epochs which means that the training is converging to an optimal point.
you can extract movie embedding from the movie_model for each movie ID using movie_model.predict(numpy.array([id])).
Now how do you know that these embeddings are correct? One way to verify them is to plot those embeddings. Once you plot, you will see that movies with a similar rating (as the rating was our target) are close by in the embedding space. Let's have a look.
Figure 5 is a little hard to visualize. So I am going to plot the 1st 300 movie embeddings to check if I can get a clearer picture.
For example, in Figure 6 — movie ID 53and 66are closer in embedding space than movie ID 9.
If you check the average rating from your dataset (Figure 7), you will observe that movie ID 53and 66has an average rating of 3.24and 3.04 respectively while movie ID 9has an average rating of 3.83.
Now that we have the embeddings, we can apply the nearest neighbor algorithm to recommend movies to users. We can get user_embedding from the user_id and find out what are the movies that are close to the user_embedding in the embedding space. We use K-Nearest Neighbor algorithm for this purpose.
A list of 10 movies recommended here is shown in Figure 8.
Something to note here is, the list of recommended movies might vary if the embedding changes. Embeddings might change run to run. This is primarily because we are using a smaller dataset (100K rows). A bigger dataset (1M+) along with longer training time is likely to output more stable embeddings.
Complete code for this article can be found here.
Deep Neural Networks for Youtube Recommendation hereKula, M. (2015). Metadata embeddings for user and item cold-start recommendations. arXiv:1507.08439.Keras Embedding layer https://keras.io/api/layers/core_layers/embedding/How does Keras Embedding layer work https://stats.stackexchange.com/questions/270546/how-does-keras-embedding-layer-workhttps://drop.engineering/building-a-recommender-system-using-embeddings-de5a30e655aahttps://www.kaggle.com/willkoehrsen/neural-network-embedding-recommendation-system#Neural-Network-Embedding-Model
Deep Neural Networks for Youtube Recommendation here
Kula, M. (2015). Metadata embeddings for user and item cold-start recommendations. arXiv:1507.08439.
Keras Embedding layer https://keras.io/api/layers/core_layers/embedding/
How does Keras Embedding layer work https://stats.stackexchange.com/questions/270546/how-does-keras-embedding-layer-work