Building a Recommendation System with PySpark MlLib — Part 1
By: Muhammad Faheem Sharif
Tags: Collaborative Filtering, Recommendation Engine, Data Science, Apache Spark, Machine Learning

Recommendation Systems and their implementation on Distributed Computation Framework Spark

Source : Marutitech

This article is Co-Authored by Ubaid-ur-Rehman and Zakriya Ali Sabir

Recommendations systems have gone a long way under evolution process since their inception back in 1994. In 1994, a recommendation system by the name of “Tapestry” was built by P. Resnick and his team at MIT. Tapestry, was designed to recommend documents from newsgroups. The authors also introduced the term collaborative filtering as they used social collaboration to help users with large volume of documents.

Since then, recommendation systems and the approaches used for their creation have gone under serious evolution, thanks to competitions like Netflix Prize competition. Firstly, let’s realize the importance of recommendation systems.

Importance of Recommendation Systems

To realize the importance of Recommendation Systems, McKinsey published a report stating 35% of Amazon’s revenue is generated by its recommendation engine.

Also, the Netflix Prize competition mentioned earlier was worth $1 million and it took 3 years to complete. In 2006, Netflix hosted a competition worth $1 million for improving their recommendation engine by 10%. It took three years to complete. Results of Netflix Prize Competition can be seen here. The significance of recommendation system is evident from the facts and figures stated above.

Besides, there is another important factor which increases the significance of recommendation system. Nowadays, with humongous amount of data and information available over the internet, it becomes a hassle for user to filter out the exact same information that he/she wants.

Recommender Systems as Mutualists

Source: Google

Recommendation system serves the “Mutualism” relationship between two parties. Mutualism is a relationship in which both the involved parties in a relationship benefit by maintaining a relationship just as a relationship between humming bird and flowers. How does this work in this context? Lets digest this by an example:

Suppose you are a visitor to e-commerce website and you buy an item there. The website starts recommending you similar items based on your behavior at that particular website. Lets say their recommendation system is 100% accurate (only in ideal scenarios) and will only recommend you items that you are highly interested in. What will happen is that you will start visiting their website more often generating more traffic to their website and ultimately revenue for them. But how this will benefit you as a user? Imagine crawling the whole web without any assistance to get to your required item. In this era of overwhelming information on the web, it is very difficult to distill your required information. Therefore, a good recommender system along with adding revenue to the provider, is a very good friend of the user and helps save time in this very fast paced world.

Recommendation System Techniques

Before moving on to algorithm explaining and implementing part, lets first discuss different approaches to achieving a recommender system.

Source: VUB Artificial Intelligence Lab

Above flow chart displays the basic techniques of implementing a recommender system from an eagle eye’s perspective.

In this article, we will be explaining an algorithm that is built on top of collaborative filtering technique. Therefore, we will discuss only collaborative filtering here.

For content-based algorithms, this type of filter does not involve other users if not ourselves. Based on what we like, the algorithm will simply pick items with similar content to recommend us. This is one of the metric that we can use when calculating similarity, between users or contents. Perhaps we will deep dive content-based algorithms more in future articles.