Building a Recommendation System with Spark ML and Elasticsearch | by Lijo Abraham | Towards Data Science

While working in search teams of different product companies, there was always a constant demand for building a recommendation system for users. We have build with popular search engines like Elasticsearch, SOLR, and tried perfecting the search relevance. But there was always a limit when it comes to doing search relevance and creating recommendation system with just search engines. I was always amazed by companies like Amazon and Netflix providing recommendations based on our interest. So by using Apache spark machine learning capabilities along with elasticsearch, we are going to build a recommendation system.
As in the picture above, if User 1 likes Item A, Item B, and Item C and User 2 likes Item B and Item C, there is a high probability that User 2 also likes Item A and can recommend Item A to User 2.
In the field of recommendations system, there are mainly three techniques used for providing recommendations.
Collaborative filtering
Content-based
Hybrid technique
We will be using the Collaborative filtering technique in Pyspark for creating a recommendation system. Apache Spark ML implements alternating least squares (ALS) for collaborative filtering, a very popular algorithm for making recommendations.
ALS is a matrix factorization algorithm that uses Alternating Least Squares with Weighted-Lambda-Regularization (ALS-WR).
The main problem with Spark ALS is that it will only recommend top products for a specific user (user-products model) and top users for a specific product (product-users model).
Calculating an all-pairs similarity is not scaling to big product catalogues. The growing number of combinations O(n2) leads very quickly to overly costly shuffle operations and infeasible computer times. For calculating the Item-Item similarity, we are building an item-item model using the similarity of item factors from the ALS model using spark and elasticsearch.
The ALS algorithm essentially factorizes two matrix — one is userFeatures and the other is itemFeatures matrix. We are doing cosine similarity on the itemFeatures rank matrix to find item-item similarity.
The dataset contains mainly two types of data from amazon.com. One is metadata, which contains the product metadata and the other is rating data of different products respective of users.
Note: The dataset can be downloaded from this site.
The spark version used is 2.4.6 and elasticsearch version is 7.9.0
Load the product dataset into Spark.Use Spark DataFrame operations to clean up the dataset.Load the cleaned data into Elasticsearch.Using Spark MLlib, train a collaborative filtering recommendation model from the rating data.Save the resulting model data into Elasticsearch.Using Elasticsearch queries, generate recommendations.
Load the product dataset into Spark.
Use Spark DataFrame operations to clean up the dataset.
Load the cleaned data into Elasticsearch.
Using Spark MLlib, train a collaborative filtering recommendation model from the rating data.
Save the resulting model data into Elasticsearch.
Using Elasticsearch queries, generate recommendations.
The below code has been used for reading the product data and convert them to a spark dataframe.
After loading data in spark, the dataframe looks like below.
Now we need to index the product data to an Elasticsearch index. For connecting elasticsearch along with spark, we need to add a JAR file to the classpath or place the file in the jars folder of spark. The JAR file needs to be downloaded based on the spark version we have.
Now that product metadata is indexed in Elasticsearch, we need to index feature vectors for every product in the model_factor column of documents based on the primary id.
Feature vectors are generated by using the ItemFeatures matrix that is produced when we run the MLib ALS algorithm on user-product-ratings data.
After loading the rating data in spark and converted to dataframe will be like below.
After evaluating the model, we have got a Root Mean Square Error value of RMSE=0.032, which is pretty good. The lower the value, better the model.
Now we have to index the item vectors back to Elasticsearch.
Now the item vector has been indexed to elasticsearch, we need to find similar products.
For finding the similarity between two items, we will be using cosine similarity functionality. Cosine similarity is a measure of similarity between two nonzero vectors of an inner product space that measures the cosine of the angle between them.
The smaller the value of cosine distance, the more similar the items.
We will be calculating cosineSimilarity score of products using script_score functionality in elasticsearch. The cosineSimilarity function calculates the measure of cosine similarity between a given query vector and document vectors.
The below query has been used for fetching similar items from elasticsearch. We are filtering with category also for better results within the category. We can change the filter options according to our needs. The script adds 1.0 to the cosine similarity to prevent the score from being negative. To take advantage of the script optimizations, provide a query vector as a script parameter.
The below code has been used for getting similar products. We are passing the ASIN of the product and the number of product recommendations we want.
display_similar('B001A5HT94', num=5)
This will give the top 5 products as result, that is similar to the product we have passed.
Not bad! The results came similar to the product we have searched for. We can always tune the ALS algorithm for better performance and accuracy.
We can also take implicit feedback parameters for tuning the algorithm. Essentially, instead of trying to model the matrix of ratings directly, this approach treats the data as numbers representing the strength in observations of user actions (such as the number of clicks, or the number of times the product bought).
Source code can be found on Github.
Collaborative filtering- https://spark.apache.org/docs/3.0.0/ml-collaborative-filtering.html
Justifying recommendations using distantly-labeled reviews and fined-grained aspects. Jianmo Ni, Jiacheng Li, Julian McAuley. Empirical Methods in Natural Language Processing (EMNLP), 2019 ( PDF)