Building a Book Recommender Engine from Scratch and Deploying it to a Web Application | by Zach Alexander | Towards Data Science

Recommendation systems, whether we like them or not, are infiltrating every aspect of our lives. From the queue of shows and movies underneath the “Because you watched” header on your Netflix profile, to the “Made for you” song lists on Spotify, we’ve entered an era where content, media, and information are tailored to our unique interests.
Although this may be common knowledge, what may be less apparent is what goes on behind the scenes to serve these recommendations to users. We often hear about “algorithms” and “machine learning” that make up the backbone of these systems, but beyond that, we typically leave the rest to imagination.
After recently taking a Recommender Systems class in graduate school, I dove into some of the linear algebra and software behind these engines. By the last two weeks of the course, I felt I had enough knowledge of the inner-workings of these systems to build a full production web application that serves book recommendations to users, utilizing PySpark’s distributive computing capabilities, PostgreSQL, Databricks, and Flask.
If interested in adapting the project, the code is available on my GitHub, and a description of my final project, including exploratory analysis and an example of the ALS algorithm in production can be found in a separate repo.
I’ve added the web application link to the end of this piece as well, but if you’d like to create an account and start rating books/receiving book recommendations, you can join here: https://zach-book-recommender.herokuapp.com/.
Now, before we take a look at my own engine, it may be helpful to provide some context of a few overarching topics about these systems. On a very basic level, recommender systems operate via machine learning algorithms. Typically, these algorithms can be classified into two categories — content-based and collaborative filtering.
Content-based methods measure the similarity of item attributes and focus on recommending other items similar to what the user likes, based on their previous actions or explicit feedback (i.e., via surveys or ratings).
Although content-based methods are quite effective in some instances, they do have their drawbacks. Collaborative filtering methods operate differently, and do their best to address some of the limitations of content-based filtering. With collaborative filtering, algorithms use similarities between users and items simultaneously to provide recommendations. Essentially, the underlying models are designed to recommend an item to User A based on interests of a similar User B.
Collaborative filtering can be achieved in myriad ways, however, one common method is to use matrix factorization and dimensionality reduction techniques. Algorithms like Alternating Least Squares (ALS), Singular Value Decomposition (SVD), and Stochastic Gradient Decent (SGD) do this quite well, and while working with large volumes of data (i.e., 1M+ data points), these algorithms in conjunction with distributive computing and parallel processing are good working solutions to build production-ready applications with large volumes of data.
If you are interested in learning more about these techniques, I’d recommend reading more in these excellent TDS pieces:
For Alternating Least Squares (ALS)
For Singular Value Decomposition (SVD)
For Stochastic Gradient Decent (SGD)
With some baseline knowledge of how these algorithms work, I’ll jump into my own project, and use of ALS to serve book recommendations to users.
Before diving into each component of building the application, I’ll outline the overall flow of how it works and the technologies used to serve recommendations to users in the Flask web app:
As we can see, it’s a pretty intricate system that relies heavily on distributed computing in Spark to process around 1 million book ratings and compute updated recommendations to users every hour. It also depends on an important connection between the deployed PostgreSQL database and the Amazon Web Services cluster that contains my ALS algorithm.
With the overall flow established above, we can now break down each step of building the engine.
As a first step, it was imperative to find a large user-book dataset that would work well with my choice of algorithms and would provide reliable predictions and recommendations to users.
Therefore, I did some exploratory analysis and decided to use the goodbooks-10k dataset. According to Kaggle, this dataset contains close to 1 million ratings across 10,000 different books located on a popular book platform, Goodreads. With this dataset in hand, I noticed a few things:
The dataset contained a total of 53,425 users that have supplied ratings for at least 10 books.
The average book rating was about 3.85, with 4 being the most common rating on a likert scale of 1 to 5. Therefore, most ratings were quite positive.
As expected, the dataset was very sparse (~99.82% of the ratings file was blank).
Some of the most highly rated books in the dataset were household favorites, including parts of the Calvin and Hobbes collection, Harry Potter, and a few books with religious principles.
To examine some of this exploratory analysis, you can go to this jupyter notebook.
Next, after doing some exploratory analysis, I decided to build out a simple python Flask application in order to allow new users to do the following:
Register/sign-in under a specific username
Load a profile page that displays a history of a user’s rated books and provides navigation to either rate more books or view recommended books (see an example below):
Incorporate a search feature that interacts with the Goodreads API to allow users to search for a book by title name, and then rate the book on a scale of 1–5 (see example below):
After rating a few books, the user can then see their individualized recommendations on a separate page (will be shown later in the article).
As an important note, these user ratings from the application were then incorporated back into the robust goodbooks dataset and eventually into the ALS algorithm. By setting up a PostgreSQL database, I was able to match new user ids, their ratings and the book ids to this original dataset. Therefore, whenever a new rating comes in from a user, it is appended to the ratings dataset which I used later for my recommender engine.
After doing a lot of testing locally to ensure that book ids and user ids matched in the database, I deployed the application with Heroku.
For instructions on how to deploy a Flask application and a corresponding PostgreSQL database with Heroku, you can refer to this extremely helpful YouTube tutorial by Brad Traversy. Specifics for deploying with Heroku can also be found in this piece.
With the application functionality set up and deployed, I then used Databricks and Pyspark to build out an Alternating Least Squares (ALS) algorithm that absorbs my ratings dataset from my PostgreSQL database, computes the updated predictions based on matrix factorization techniques established in Databricks, and then recommends books to each user based on the highest predicted scores per user.
Here’s a bit of code below that outlines this process:
First, I set up a schema for a Spark dataframe before I read in the ratings table from PostgreSQL:
from pyspark.sql.types import StructType, StructFieldfrom pyspark.sql.types import DoubleType, IntegerType, StringTyperatings_schema = StructType([  StructField("col_id", IntegerType()),  StructField("userid", IntegerType()),  StructField("rating", DoubleType()),  StructField("book_id", IntegerType()),  StructField("username", StringType()),  StructField("isbn10", StringType())])
Then, after setting up our environment variables and a JDBC connection to my remote PostgreSQL database, I read in the table and saved it as a Spark dataframe:
remote_table = spark.read.format("jdbc")\  .option("driver", driver)\  .option("inferSchema", ratings_schema) \  .option("url", url)\  .option("dbtable", table)\  .option("user", user)\  .option("password", password)\  .load()
With the Spark dataframe loaded into Databricks, I then started to set up the data to run my ALS model. First, I split the dataframe into a training, validation and testing dataset — 60% training, 20% validation and 20% testing. The validation dataset was used to test the fine-tuning of my model parameters, and allowed me to have a hold-out test set that I used to compute the Root Mean Square Error (RMSE) on the final, optimized model.
(training, validation, test) = remote_table.randomSplit([0.6, 0.2, 0.2])  # caching data to cut down on cross-validation time later training.cache() validation.cache() test.cache()
With the data split, I did some fine-tuning of the model to determine the optimal parameters. Then, I ran the ALS model with these optimal parameters, and fit this to the training dataset.
als = ALS(maxIter=10, regParam=0.20, userCol="userid", itemCol="book_id", ratingCol="rating", coldStartStrategy="drop", nonnegative = True, implicitPrefs = False).setRank(50)model = als.fit(training)
You’ll notice, that I set the rank of the matrix to 50, which cut down on processing time and still yielded valuable predictions. I then was able to use this model to compute predictions on the test dataset and determine the RMSE value.
predictions = model.transform(test)rmse = evaluator.evaluate(predictions)
In the end, I was able to get the RMSE value around 0.9, which for our purposes, and a likert scale of 1–5, isn’t too bad. I then was able to use this model to generate predictions and recommendations for the full ratings dataset. To do this, I used the recommendForAllUsers() function in PySpark, to find the 10 highest rated books for each user.
ALS_recommendations = model.recommendForAllUsers(numItems = 10)# Temporary table ALS_recommendations.registerTempTable("ALS_recs_temp") clean_recs = spark.sql("""SELECT userid,                             bookIds_and_ratings.book_id AS book_id,                             bookIds_and_ratings.rating AS prediction                         FROM ALS_recs_temp LATERAL VIEW explode(recommendations) exploded_table AS bookIds_and_ratings""")clean_recs.join(remote_table, ["userid", "book_id"], "left").filter(remote_table.rating.isNull()).show()clean_recs_filtered = clean_recs.select("userid", "book_id", "prediction")
After creating a temporary table to re-orient the recommendations, I then wanted to ensure that books that a user in the application rated (and implicitly read), would not be recommended back to them in the app. Therefore, I did some additional cleaning to only recommend new books to users:
new_books = (clean_recs_filtered.join(remote_table, ["userid", "book_id"], "left").filter(remote_table.rating.isNull()))
As a final step, I identified only user ids that had signed up via my application to append recommendations back to the PostgreSQL database. Because there was such a large number of predictions and ratings from the initial goodbooks dataset that would go back to nonexistent users, I filtered to only include recommendations for users that joined through the web application:
new_books_fnl = new_books.select('userid', 'book_id', 'prediction')new_books_users = new_books_fnl.filter(new_books_fnl['userid'] > 53424)new_books_use = new_books_users.select('userid', 'book_id', 'prediction')
And here’s an example of recommendations served back to users that have joined the web application:
We now have recommendations for users in the web app!
As we can see, some of the predictions aren’t very good — for instance the first user in the table will be served a book that the algorithm predicted they would rate around a 3.5/4. Ideally we want to find books that users will really like (without thinking about components such as serendipity, novelty, etc.). Therefore, with more time and a broader user base, I hope to see recommended books with higher predicted ratings.
With the model working locally in Databricks, I then was able to upload this optimized model and deploy it to an Amazon Web Services cluster, running a cronjob every hour to process any new ratings that are absorbed into my PostgreSQL database. The hourly cronjob essentially will re-run the model, re-compute the predictions and recommendations, and then overwrite a recommendations table in my PostgreSQL with the updated recommendations. This is then connected back to my Flask app to serve to users. For deployment instructions related to AWS and Databricks, you can find some documentation here.
With the recommendations table in my PostgreSQL updated with my recommendations and their corresponding book ids, I could then utilize the Goodreads API to find the book information (i.e. picture, title, etc.) to build out on the front-end application.
In the end, I did some testing and noticed that the recommendations weren’t too bad! For instance, I created a few users that were data science fanatics, and rated a lot of Data Science books. I intentionally had some overlap in ratings of books across users, but also tried to rate a few new books for each user.
When I ran my model to serve up recommendations, I was pleased to see that books that were not rated by one data science fanatic were being recommended to them based on positive ratings from another data science fanatic, which demonstrates that my collaborative filtering technique was working. See an example below:
As we can see from the titles boxed in red, User 2 rated those two books very highly (rating of 5). Therefore, when running our machine learning algorithm, there was some overlap between User 1 and User 2’s book ratings — which meant that the books that weren’t rated by User 1 (and implicitly not read), but were rated highly by User 2, were then served as recommendations to User 1.
To give the application a try, go here: https://zach-book-recommender.herokuapp.com/
Although recommendations for data science books do seem to work pretty well, there is a large gap in other genres and topics. You’ll probably notice that your recommendations will also pull in other very highly rated books from the goodbooks-10k dataset (i.e. Calvin and Hobbes, etc.), until more users that are similar to your tastes join the app. Therefore, I’d recommend sharing it with friends, and see if over time, your recommendations start to get better and better!
If you enjoyed this piece, or found it helpful, please get in touch! I love collaborating with others on projects and learning from peers that are passionate about data science work.
You can also check out my data visualization portfolio, where I primarily work with Javascript and D3.js, and talk more about other projects I’ve done in grad school.