Building a Book Recommender Engine from Scratch and Deploying it to a Web Application
By: Zach Alexander
Tags: Postgres, Data Science, Machine Learning, Book Recommendations, Spark

Working with Databricks and PySpark — developing my ALS algorithm

With the application functionality set up and deployed, I then used Databricks and Pyspark to build out an Alternating Least Squares (ALS) algorithm that absorbs my ratings dataset from my PostgreSQL database, computes the updated predictions based on matrix factorization techniques established in Databricks, and then recommends books to each user based on the highest predicted scores per user.

Here’s a bit of code below that outlines this process:

First, I set up a schema for a Spark dataframe before I read in the ratings table from PostgreSQL:

from pyspark.sql.types import StructType, StructField

from pyspark.sql.types import DoubleType, IntegerType, StringType



ratings_schema = StructType([

StructField("col_id", IntegerType()),

StructField("userid", IntegerType()),

StructField("rating", DoubleType()),

StructField("book_id", IntegerType()),

StructField("username", StringType()),

StructField("isbn10", StringType())

])

Then, after setting up our environment variables and a JDBC connection to my remote PostgreSQL database, I read in the table and saved it as a Spark dataframe:

remote_table = spark.read.format("jdbc")\

.option("driver", driver)\

.option("inferSchema", ratings_schema) \

.option("url", url)\

.option("dbtable", table)\

.option("user", user)\

.option("password", password)\

.load()

With the Spark dataframe loaded into Databricks, I then started to set up the data to run my ALS model. First, I split the dataframe into a training, validation and testing dataset — 60% training, 20% validation and 20% testing. The validation dataset was used to test the fine-tuning of my model parameters, and allowed me to have a hold-out test set that I used to compute the Root Mean Square Error (RMSE) on the final, optimized model.

(training, validation, test) = remote_table.randomSplit([0.6, 0.2, 0.2]) # caching data to cut down on cross-validation time later training.cache()

validation.cache()

test.cache()

With the data split, I did some fine-tuning of the model to determine the optimal parameters. Then, I ran the ALS model with these optimal parameters, and fit this to the training dataset.

als = ALS(maxIter=10, regParam=0.20, userCol="userid", itemCol="book_id", ratingCol="rating", coldStartStrategy="drop", nonnegative = True, implicitPrefs = False).setRank(50) model = als.fit(training)

You’ll notice, that I set the rank of the matrix to 50, which cut down on processing time and still yielded valuable predictions. I then was able to use this model to compute predictions on the test dataset and determine the RMSE value.

predictions = model.transform(test)

rmse = evaluator.evaluate(predictions)

In the end, I was able to get the RMSE value around 0.9, which for our purposes, and a likert scale of 1–5, isn’t too bad. I then was able to use this model to generate predictions and recommendations for the full ratings dataset. To do this, I used the recommendForAllUsers() function in PySpark, to find the 10 highest rated books for each user.

ALS_recommendations = model.recommendForAllUsers(numItems = 10) # Temporary table ALS_recommendations.registerTempTable("ALS_recs_temp") clean_recs = spark.sql("""SELECT userid, bookIds_and_ratings.book_id AS book_id, bookIds_and_ratings.rating AS prediction FROM ALS_recs_temp LATERAL VIEW explode(recommendations) exploded_table AS bookIds_and_ratings""") clean_recs.join(remote_table, ["userid", "book_id"], "left").filter(remote_table.rating.isNull()).show()



clean_recs_filtered = clean_recs.select("userid", "book_id", "prediction")

After creating a temporary table to re-orient the recommendations, I then wanted to ensure that books that a user in the application rated (and implicitly read), would not be recommended back to them in the app. Therefore, I did some additional cleaning to only recommend new books to users:

new_books = (clean_recs_filtered.join(remote_table, ["userid", "book_id"], "left").filter(remote_table.rating.isNull()))

As a final step, I identified only user ids that had signed up via my application to append recommendations back to the PostgreSQL database. Because there was such a large number of predictions and ratings from the initial goodbooks dataset that would go back to nonexistent users, I filtered to only include recommendations for users that joined through the web application:

new_books_fnl = new_books.select('userid', 'book_id', 'prediction')



new_books_users = new_books_fnl.filter(new_books_fnl['userid'] > 53424)



new_books_use = new_books_users.select('userid', 'book_id', 'prediction')

And here’s an example of recommendations served back to users that have joined the web application:

Snapshot of the books that’ll be recommended back to users in the web application | Image by author

We now have recommendations for users in the web app!

As we can see, some of the predictions aren’t very good — for instance the first user in the table will be served a book that the algorithm predicted they would rate around a 3.5/4. Ideally we want to find books that users will really like (without thinking about components such as serendipity, novelty, etc.). Therefore, with more time and a broader user base, I hope to see recommended books with higher predicted ratings.

With the model working locally in Databricks, I then was able to upload this optimized model and deploy it to an Amazon Web Services cluster, running a cronjob every hour to process any new ratings that are absorbed into my PostgreSQL database. The hourly cronjob essentially will re-run the model, re-compute the predictions and recommendations, and then overwrite a recommendations table in my PostgreSQL with the updated recommendations. This is then connected back to my Flask app to serve to users. For deployment instructions related to AWS and Databricks, you can find some documentation here.

With the recommendations table in my PostgreSQL updated with my recommendations and their corresponding book ids, I could then utilize the Goodreads API to find the book information (i.e. picture, title, etc.) to build out on the front-end application.

So, How Well is it Working?

In the end, I did some testing and noticed that the recommendations weren’t too bad! For instance, I created a few users that were data science fanatics, and rated a lot of Data Science books. I intentionally had some overlap in ratings of books across users, but also tried to rate a few new books for each user.

When I ran my model to serve up recommendations, I was pleased to see that books that were not rated by one data science fanatic were being recommended to them based on positive ratings from another data science fanatic, which demonstrates that my collaborative filtering technique was working. See an example below: