Building a recommendation engine to recommend books in Spark | by Saket Garodia | Towards Data Science

Using collaborative filtering to predict ratings of unread books on ‘Goodreads’
Warren Buffett was once asked about the key to success, he pointed to a stack of nearby books and said, “Read 500 pages like this every day. That’s how knowledge works. It builds up, like compound interest. All of you can do it, but I guarantee not many of you will do it.”
Books are the best resources for most of us to develop and gain perspectives. I myself love reading books. Once I like a book, I have a habit of going to good books or asking someone who has a similar taste to look for recommendations for the next series of books I might like.
Artificial Intelligence has made our world so easy by recommending us books, movies, and products all based on the past data that saves our time and energy on analyzing different options. In fact, sometimes machines can recommend us better than what we think because they don’t suffer from emotional biases.
In this blog, I will explain how I used the ‘goodbooks ’ data to build a recommendation engine in Spark. I used the Alternating Least Square(ALS) algorithm from the Spark library which is basically the implementation of the paper published by the Netflix competition winners. Before delving into the implementation let me try to explain the ALS algorithm.
If you want to move ahead directly to the ‘goodbooks ’problem, kindly skip this section.
For someone who loves reading academic papers, here’s the link to the paper: https://datajobs.com/data-science-repo/Recommender-Systems-%5BNetflix%5D.pdf.
To understand the different types of recommender systems, kindly visit my other blogs where I have explained the different kinds of recommender systems like content-based, metadata-based, and collaborative filtering based approaches. Here’s the link to those series of blogs: https://medium.com/analytics-vidhya/the-world-of-recommender-systems-e4ea504341ac?source=friends_link&sk=508a980d8391daa93530a32e9c927a87
Basically, a content-based recommender system means recommending some products based on the content of previous products. If you read a book say ‘Think and Grow Rich’, you will be recommended other books having similar content (or text in this case )like ‘Awaken The Giant Within’ since they both are self-help books and they might contain similar content.
The most important kind of recommender system is collaborative filtering based approach. Let’s say you know a friend who has the same taste as you because you both love psychology, then you might like reading other books that your friend has read but you haven’t. This is the sole concept behind collaborative filtering. Now let's talk specifically about Alternating least square method.
Collaborative filtering can be easily achieved by matrix factorization techniques like Singular Value decomposition where a user-rating matrix is decomposed into the user-concept matrix, concept-weights matrix, and rating-concept matrix. Concepts are basically latent or hidden factors that the matrix decomposition implicitly generates like in the case of the books, the different concepts can be psychology, data science, philosophy, etc.
For any given item, just a basic dot product can give a rating but there’s one problem with this approach.
Most of the matrix factorization techniques like Singular Value decomposition don’t know how to deal with an incomplete/sparse matrix which means having empty values in the user-rating matrix(which is common as not every user would have read all the books). Traditionally, engineers have been imputing those values with the mean or median before performing matrix factorization to do collaborative filtering. This leads to overfitting since the books that have never been rated are being imputed by the mean or median which can skew the results towards them.
Recent methods like Alternating Least square don’t suffer from these fallbacks. They suggest modeling directly the observed ratings while avoiding overfitting through a regularized model. To learn the factor vectors (pu and qi ), the system minimizes the regularized squared error on the set of known ratings:
They basically try to minimize the error just like gradient descent. In the above equation, the intuition is that r is the given rating and qi*pu is basically the dot product of the matrices ( results from SVD ) to predict rating and we are trying to minimize it but there’s another challenge to minimize the above equation. There are 2 unknowns to minimize, so it's not a convex optimization problem that leads to a global minimum.
ALS rectifies this issue while providing some other benefits too. ALS techniques rotate between fixing the qi ’s and fixing the pu ’s. When all pu ’s are fixed, the system recomputes the qi ’s by solving a least-squares problem and vice versa. This ensures that each step decreases the above equation until convergence. This gives rise to potentially massive parallelization of the algorithm and therefore can be easily implemented in Spark. Now, let's move to our good books problem.
The ‘goodbooks ’ dataset contains about 1 million ratings across 10000 different books. In most cases, there are at least 10 books rated by each user and the rating lies between 0 and 5. We will use these 1 million ratings to predict ratings for all the other books that haven’t been read by the users. Also, there is a ‘to_read ’ dataset that contains the details of the books that a user has kept to read in the future (like wishlist). After building our recommendation system, we will try to predict the rating of these ‘to_read’ books to see how our algorithm performs on these. The ratings predicted on these to_read books should be high since the users have already wished to read these books.
Here’s the link to the dataset:
www.kaggle.com
I will use ‘databricks ’ community edition since it is the best platform to run ML on spark and its free.
Let's look at the ratings and books’ data frames. The ‘ratings ’ data contain book_id, user_id, and rating. The books dataset is like metadata to all the 10000 books containing the book name, the number of ratings, book_id, and URLs to the image of the books cover. I found it fun to print the book cover in the ‘pandas ’ data frame.
Since I love reading books, I did some exploration on the books to know how old these books are based on their publication year, whats their average ratings, what are the top-rated books and the books with the most number of reviews.
I saw some of the books were also present from ‘Before Christ’ like this one. The Epic of Gilgamesh’s publication year is 1750 BC. Interesting. Isn’t it?
Let’s see the distribution of the number of books published after 1950.
%sqlselect original_publication_year, count(*) as count from books where original_publication_year > 1950 group by original_publication_year
Well, after 1990, each year seems to have about 100 books and then it increases with each year until it reaches its peak.
Let’s see the books with the most number of ratings.
 most_ratings = books_df.sort_values(by = ‘ratings_count’, ascending = False)[[‘original_title’,’ratings_count’, ‘average_rating’, ‘image_url’ ]][0:10]import pandas as pdfrom IPython.display import Image, HTMLmost_ratings[‘img_html’] = most_ratings[‘image_url’]\ .str.replace( ‘(.*)’,  ‘<img src=”\\1" style=”max-height:124px;”></img>’ )with pd.option_context(‘display.max_colwidth’, 10000):  display(HTML(most_ratings[[‘original_title’, ‘img_html’, ‘ratings_count’, ‘average_rating’ ]].to_html(escape=False)))
We can see that books with most numbers of ratings are famous books like The Hunger Games, Harry Potter, etc.
Now, lets the books ranked by the average ratings ( without considering the number of ratings).
high_rating_books = books_df.sort_values(by = ‘average_rating’, ascending = False)[[‘original_title’,’ratings_count’,’image_url’, ‘average_rating’ ]][0:10]high_rating_books[‘img_html’] = high_rating_books[‘image_url’]\ .str.replace( ‘(.*)’,  ‘<img src=”\\1" style=”max-height:124px;”></img>’ )with pd.option_context(‘display.max_colwidth’, 10000):  display(HTML(high_rating_books[[‘original_title’, ‘img_html’,’ratings_count’, ‘average_rating’ ]].to_html(escape=False)))
Well, the complete Calvin and Hobbes and the Words of Radiance have the highest average ratings among the 10000 books.
Let’s see the authors with the most number of published books:
authors_with_most_books = pd.DataFrame(books_df.authors.value_counts()[0:10]).reset_index()authors_with_most_books.columns = [‘author’, ‘number_of_books’]
Stephen King has 60 books attributed to his name in the dataset.
Now, let’s see the distribution of average ratings of all the 10000 books:
import matplotlib.pyplot as pltplt.figure(figsize=(12,6))plt.title(‘Distribution of Average Ratings’)books_df[‘average_rating’].hist()displaay()
In the rating ‘dataframe ’ which has about a million ratings, each book has about 100 ratings, and its a pretty unbiased dataset. There are very few books with less than 100 ratings.
Now, let’s see the sparsity of the users-rating matrix to understand how much biases could have been created if we wouldn't use ALS and use traditional SVD for collaborative filtering.
# Count the total number of ratings in the datasetnumerator = ratings.select(“rating”).count()# Count the number of distinct Id’snum_users = ratings.select(“user_id”).distinct().count()num_items = ratings.select(“book_id”).distinct().count()# Set the denominator equal to the number of users multiplied by the number of itemsdenominator = num_users * num_items# Divide the numerator by the denominatorsparsity = (1.0 — (numerator * 1.0)/ denominator) * 100print(“The ratings dataframe is “, “%.2f” % sparsity + “% empty.”)
The rating data frame is 99.82% empty.
Of course, it can't be possible that every unique user has rated all the books therefore it's better to build an algorithm only on the basis of the original ratings. Enter the ALS world.
Now, let’s split the data into training and test set to use collaborative filtering using Alternate Least Square method.
(training, test) = ratings.randomSplit([0.8, 0.2])
Let’s import ALS and regression evaluator to find RMSE.
from pyspark.ml.evaluation import RegressionEvaluatorfrom pyspark.ml.recommendation import ALSfrom pyspark.sql import Rowals = ALS( userCol=”user_id”, itemCol=”book_id”, ratingCol=”rating”, coldStartStrategy=”drop”, nonnegative = True, implicitPrefs = False)
‘implicitPrefs ’ argument is used when we are not using explicit data like the rating data here. Sometimes companies don't have explicit data like ratings and still want to build a recommendation engine using other proxies like views, clicks, wishlists, etc. In that case, implicit preference is used but is out of the scope of our good books project. ‘coldStartStrategy ’ is used when we don't have any data for a user which might lead to null prediction if the user on the test set has no rating in the training set. We have dropped the cold start strategy because we want to avoid such situations for our problem in hand.
Now let's build our hyperparameter list and then fit the algorithm on the training data.
from pyspark.ml.tuning import ParamGridBuilder, CrossValidatorparam_grid = ParamGridBuilder() \ .addGrid(als.rank, [10, 50, 75, 100]) \ .addGrid(als.maxIter, [5, 50, 75, 100]) \ .addGrid(als.regParam, [.01, .05, .1, .15]) \ .build()# Define evaluator as RMSEevaluator = RegressionEvaluator(metricName = “rmse”,  labelCol = “rating”,  predictionCol = “prediction”)# Print length of evaluatorprint (“Num models to be tested using param_grid: “, len(param_grid))
Num models to be tested using param_grid: 64
There is a total of 64 models that will be tested and tuned before we receive the final model. The power of parallelization and spark will make it pretty fast though.
# Build cross validation using CrossValidatorcv = CrossValidator(estimator = als,  estimatorParamMaps = param_grid,  evaluator = evaluator,  numFolds = 5)model = als.fit(training)predictions = model.transform(test)predictions.show(n = 10)
Well, the prediction on the test set shows that it is very close to the original rating. Like for example the rating for user_id 14372 was originally 3 and our algorithm predicted it to be 3.13 which is pretty close.
rmse = evaluator.evaluate(predictions)print(“Root-mean-square error = “ + str(rmse))
Root-mean-square error = 0.913, On average the mean error is .9 that is the difference between the original rating and the predicted rating. Now, let’s predict 10 books and ratings for each user.
# Generate n recommendations for all usersALS_recommendations = model.recommendForAllUsers(numItems = 10) # n — 10ALS_recommendations.show(n = 10)# Temporary tableALS_recommendations.registerTempTable("ALS_recs_temp")clean_recs = spark.sql("""SELECT user_id,                            movieIds_and_ratings.book_id AS book_id,                            movieIds_and_ratings.rating AS prediction                        FROM ALS_recs_temp                        LATERAL VIEW explode(recommendations) exploded_table                            AS movieIds_and_ratings""")clean_recs.show()# Recommendations for unread books(clean_recs.join(ratings, ["user_id", "book_id"], "left")    .filter(ratings.rating.isNull()).show())new_books = (clean_recs.join(ratings, ["user_id", "book_id"], "left")    .filter(ratings.rating.isNull()))
Now, let’s use the ‘to_read ’ data( wishlist data) and see what will our algorithm predict for the books that are in the wishlist.
It should be pretty good since the users kept them in their wishlist.
recommendations = new_books.join(to_read, on = [“user_id”, “book_id”],  how = “inner”)print(recommendations.show())(recommendations     .withColumn('pred_trunc', recommendations.prediction.substr(1,1))     .groupby('pred_trunc')     .count()     .sort('pred_trunc')    .show())import matplotlib.pyplot as pltplt.figure(figsize=(12,6))plt.title('Most of the predicted ratings are above 3.8', fontsize = 12)plt.suptitle('Distribution of predictedd ratings for the to_do lists', fontsize = 18)rec = recommendations.toPandas()rec['prediction'].hist()display()
Most of the books in the wishlist are predicted as 4.5 or more which is expected.
You can access my code here:-
https://github.com/garodisk/Goodreads-recommendation-engine
Thanks a lot for reading!
References:
www.kaggle.com
spark.apache.org
Contextual multi-armed bandit — (Intuition behind Netflix Artwork Recommendation )