Crafting Recommendation Engine in PySpark
By: Shashwat Tiwari
Tags: Machine Learning, Pyspark, Spark, Recommender Systems, Big Data

Hello,

In Today’s Online Era every customer is loaded with multiple choices. Suppose recently I was looking to buy a budget-friendly laptop without having any idea of what to buy. There are possibilities that might waste a lot of time browsing around on the internet and crawling through various sites hoping to find some useful info. I might look for recommendations from other people.

The rapid growth of data collection has led to a new era of information. Data is being used to create more efficient systems and this is where Recommendation Systems comes into play. Recommendation Systems are a type of information filtering system as they improve the quality of search results and provide items that are more relevant to the search item or related to the search history of the user. They are used to predict the rating or preference that a user would give to an item. Companies like Netflix and Spotify depend highly on the effectiveness of their recommendation engines for their business and success

There are might be some website or web app that might recommend me some best laptops based on my previous purchases or laptop owned by my friends. I could just log in and jackpot! I got well-recommended laptops matched to my taste.

This is the power that is being harnessed by Recommendation Engines. From Amazon to Netflix, recommendation engines are one of the most widely used applications of Machine learning.

This Blog basically dedicated to Intuitive Explanation of Recommendation engines and its implementation in PySpark.

So Let’s Roll!

Recommendation Engines

Recommendation Engines better known as recommender system is a tool that analyzes data in order to make suggestions for something that might be a website, Ad banner, products user might be interested in.

Putting it in a more simple fashion, It is basically a system that suggests services, products, information to the user based on their trends or analysis of data. This data can be the history of the user or behavior of similar users.

The recommendation system opens the door for users to expose to the whole digital world through their experiences, behaviors, preferences, and interests. These systems also provide an efficient way for companies like Amazon, Walmart to provide consumers with personalized information and solutions.

Advantages of Recommender Systems

These systems are helpful in increasing click-through rates, revenues significantly for organizations

Recommendation engines empower positives effects on user experiences hence bringing higher customer satisfaction and retention.

For example, With Netflix Instead of having to browse through thousands of box sets and movie titles, Netflix presents you with a much narrower selection of items that you are likely to enjoy. This functionality delivers a better user experience. With this, Netflix achieved lower cancellation rates, saving the company around a billion dollars a year.

Amazon Started using the Recommender system almost 20 years ago Now it has been deployed to other industries such as finance and travel.

Workflow for Recommendation Systems

Basically Recommender system works by providing suggestions to a particular user through a kind of filtering process that is fully based on user perspective and browsing history. The input for the Recommender system will be information about the user like his browsing history, preferences and likings. This information reflects the prior usage of the product as well as the assigned ratings. They also rank the products and services by how many users like them. It helps in the sorting products which are liked by users the most. With the help of cosine similarity, one can also find how similar different products are to each other. If products are very similar to each other, they might appeal to the same users.

Typical Recommendation Engine Source: https://new.hindawi.com/journals/sp/2019/5941096/fig2/

There are mainly three kinds of recommender systems:-

1)Demographic Filtering- They offer generalized recommendations to every user, based on movie popularity and/or genre. The System recommends the same movies to users with similar demographic features. Since each user is different, this approach is considered to be too simple. The basic idea behind this system is that movies that are more popular and critically acclaimed will have a higher probability of being liked by the average audience. When we open some app on our phone it asks to access our location. Based on our location and people around it will recommend the things which we may like and this is a basic recommender present in any application.

2)Content-Based Filtering- They suggest similar items based on a particular item. This system uses item metadata, such as genre, director, description, actors, etc. for movies, to make these recommendations. The general idea behind these recommender systems is that if a person likes a particular item, he or she will also like an item that is similar to it. This can be seen in applications like Netflix, Facebook Watch, etc.which recommend the next movie or video based on the director, hero, etc.

3)Collaborative Filtering- This system matches persons with similar interests and provides recommendations based on this matching. Collaborative filters do not require item metadata like its content-based counterparts. This is the most sophisticated personalized recommendation that means it takes into account what user likes and not likes. The main example of this is Google Ads. Under the umbrella of Collaborative filtering, we have following kind of methods-

Memory Based - It basically identifies the clusters of users in order to calculate the interactions of one specific user to predict the interactions of other similar users. The second thought process will be identifying the items clusters rated by user A and predicting user’s interaction with item B.Memory based methods fail while dealing with large sparse matrices.

It basically identifies the clusters of users in order to calculate the interactions of one specific user to predict the interactions of other similar users. The second thought process will be identifying the items clusters rated by user A and predicting user’s interaction with item B.Memory based methods fail while dealing with large sparse matrices. Model Based-Methods basically revolves around ML and Data mining tools and techniques. The traditional Machine learning approach is used to train models and getting prediction out of it. One advantage of these methods is that they are able to recommend a larger number of items to a larger number of users, compared to other methods like memory-based. These methods work well with large sparse matrices as compared to the memory-based approach.

3) Hybrid Systems -Consolidated both types of information with the aim of avoiding problems that are generated when working with one kind of Recommender systems.

Now Enough of Theories Practical time Now!

Building Recommendation Engine with PySpark

According to the official documentation for Apache Spark -

“Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.”

On a high level, It is an open-source wide range data preprocessing engine that has its own development APIs which have distributed workloads on accomplishing streaming, machine learning or SQL workloads that demand repeated access to data sets. Spark core power lies in processing both batch and stream jobs.

Basically, Apache Spark offers high-level APIs to users, such as Java, Scala, Python, and R. Although, Spark is written in Scala still offers rich APIs in Scala, Java, Python, as well as R.

I can keep talking about Spark the whole day because of its awesomeness but it will be out of scope for this blog. For More insights Please Refer to this link.

Let’s jump straight to building a Recommendation engine using MLib Spark Python API.

Integrating PySpark with Anaconda

That's going to be a bit tricky when you are setting up Apache Spark at the Initial Stage. This Post really helps in quickly initializing with PySpark with Jupyter notebook integration.

Dataset

In order to build our recommendation system, we have used the MovieLens Dataset. You can find the movielens_rating.csv that we have used in our Recommendation System Project here. This data consists of 105339 ratings applied over 10329 movies.

Approach for Recommender System

With Collaborative filtering, we make predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating). The underlying assumption is that if a user A has the same opinion as a user B on an issue, A is more likely to have B’s opinion on a different issue x than to have the opinion on x of a user-chosen randomly.

At first, people rate different items (like videos, images, games). Then, the system makes predictions about a user’s rating for an item not rated yet. The new predictions are built upon the existing ratings of other users with similar ratings with the active user.

Importing Essential Libraries

In order to build the Machine learning model with PySpark, we will use the following packages-

findSpark-For Jupyter Notebook Integration with Spark

spark.Session-Entry point for any Spark Application.

pyspark.ml- Machine Learning API for PySpark.

pandas-For Data import and EDA.

Confused about Spark Session let’s break it down pyspark.sql.SparkSession(sparkContext, jsparkSession=None) The entry point to programming Spark with the Dataset and DataFrame API. A SparkSession can be used to create DataFrame, register DataFrame as tables, execute SQL over tables, cache tables, and read parquet files. In order to create Spark Session follow below build pattern-

Retrieving the Data & Data-Preprocessing

For Spark, 2.x comes with great power of inbuilt loading of .csv file as a PySpark Dataframe. We will use the data.describe().show() function to display information about the spark data frame.

Output Screenshot:

We can do a split to evaluate how well our model performed, but keep in mind that it is very hard to know conclusively how well a recommender system is truly working for some topics. Especially if subjectivity is involved, for example not everyone that loves star wars is going to love star trek, even though a recommendation system may suggest otherwise.

As we are using a Smaller dataset so 80:20 train split will be sufficient enough for our use-case.

(training, test) = data.randomSplit([0.8, 0.2])

Build the recommendation model using ALS on the training data

Spark provides a machine learning algorithm MLib.Spark MLib algorithm aims in providing practical machine leaning with scalability & ease. Below are some terminologies related to spark machine learning library-

Extraction: Extracting features from “raw” data.

Extracting features from “raw” data. Transformation : Scaling, converting, or modifying features.

: Scaling, converting, or modifying features. Selection : Selecting a subset of a larger set of features.

: Selecting a subset of a larger set of features. Locality Sensitive Hashing (LSH): This class of algorithms combines aspects of feature transformation with other algorithms.

Spark MLlib library for Machine Learning provides a Collaborative Filtering implementation by using Alternating Least Squares. The implementation in MLlib has these parameters:

numBlocks is the number of blocks used to parallelize computation (set to -1 to auto-configure).

rank is the number of latent factors in the model.

iterations are the number of iterations to run.

lambda specifies the regularization parameter in ALS.

implicitPrefs specifies whether to use the explicit feedback ALS variant or one adapted for implicit feedback data.

alpha is a parameter applicable to the implicit feedback variant of ALS that governs the baseline confidence in preference observations.

Now let's build our ALS model and fit this model with training features created by randomsplit() function.

Point to be noted here that I am not using cold start strategy which is used to ensure we don’t get NaN evaluation metrics.

Bingo!!! Our Model is fit Now Next Step will be generating Recommendation or prediction from fitted Alternating least-squares Model.

Generating Predictions & Model Evaluation

After successful execution of spark jobs its time to evaluate the build model using inbuilt transform function. This function is more or less similar to predict() function in the traditional machine learning algorithm(Sklearn). However, transform () function transform the input test data or unseen data in order to generate predictions.

Output Screenshot

Evaluating a model is a core part of building an effective machine learning model. In PySpark we will be using RMSE(Root mean squared Error) as our evaluation metric.

The RMSE described our error in terms of the rating column.

So What’s Next Still we didn't get Recommendations of Movies for our users. Let’s kick this now!

Recommending Movies with ALS

So now that we have the model, how would you actually supply a recommendation to a user?

The approach here will be simple We will be taking a single userid example11 as features and pass it to trained ALS Model. The same way we did with the test data!

Output Screenshot

Now we will use model.transform() function in order to generate recommended movies along with their predicted features.

Output Screenshot

End Notes

From the Usecase, we observe that including a recommendation in your current system will be efficient and attractive as it enforces experience to involve more customer engagement. From a business point of view, it generates great revenues. Moreover, Recommendation Systems are entirely dependent on Business goals.

PySpark has definitely powerful Machine leaning API which is continuously evolving. I strongly recommend you to Apply this Collabaortave filtering with other Recommendation engine Datasets. Keep Practicing Keep Learning!

The full code along with the Dataset used for this project is available on my GitHub.

References

If you like this post, please follow me. If you have noticed any mistakes in the way of thinking, formulas, animations or code, please let me know.

Cheers!