Recommendation system in python using ALS algorithm and Apache Spark
By: Neha Patel
Tags: Python, Data Science, Recommendation System, Apache Spark

Recommendation system has become integral part of many online platforms given it’s use in increasing sales and customer retention.Amazon, Netflix, Facebook, Linkedin and many more make use of recommendation system on their platform to recommend products,movies,people,posts respectively to users.Ever wondered how these companies provides recommendations.

Given it’s popularity there are three main techniques used for providing recommendations online-Collaborative filtering,Content-based and Hybrid technique.Here we will be making use of Alternating least square matrix factorization method,a collaborative filtering algorithm.

We are going to build a recommendation system in python using Apache spark and Jupyter Notebook.You can make this simple recommendation model as a mini project in college or as a personal project.

Requirements:

Your machine should have latest version of Python,Apache Spark and Jupyter Notebook installed. You also need to connect pyspark with Jupyter Notebook and many tutorials are available out there to do the same.

Dataset used is downloaded from the link given below:

The data used here is of Musical Instruments for training the model.

Terminologies:

There are certain terminologies which needs to be understood before moving forward.

Apache Spark: Apache Spark is an open-source distributed general-purpose cluster-computing framework.It can be used with Hadoop too. Collaborative filtering: Collaborative filtering is a method of making automatic predictions (filtering) about the interests of a user by collecting preferences or taste information from many users. Consider example if a person A likes item 1, 2, 3 and B like 2,3,4 then they have similar interests and A should like item 4 and B should like item 1. Alternating least square(ALS) matrix factorization: The idea is basically to take a large (or potentially huge) matrix and factor it into some smaller representation of the original matrix through alternating least squares. We end up with two or more lower dimensional matrices whose product equals the original one.ALS comes inbuilt in Apache Spark. PySpark: PySpark is the collaboration of Apache Spark and Python. PySpark is the Python API for Spark.

So let’s start making our recommendation model in jupyter notebook.

1.Initialize spark session:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('Recommendation_system')

.getOrCreate()

2. Load Dataset in Apache Spark

df = spark.read.json("Musical_Instruments_5.json")

df.show(100,truncate=True)

Your dataframe will look like following:

Original dataframe

3. Select appropriate columns

nd=df.select(df['asin'],df['overall'],df['reviewerID'])

nd.show()

We do not need all the columns present in the dataframe .Only asin which is ProductID,reviewerID and overall(rating given by users to each product) is required.

4. Importing important modules

from pyspark.ml.evaluation import RegressionEvaluator

from pyspark.ml.recommendation import ALS

5. Converting String to index

Before making an ALS model it needs to be clear that ALS only accepts integer value as parameters. Hence we need to convert asin and reviewerID column in index form.

from pyspark.ml.feature import StringIndexer

from pyspark.ml import Pipeline

from pyspark.sql.functions import col indexer = [StringIndexer(inputCol=column, outputCol=column+"_index") for column in list(set(nd.columns)-set(['overall'])) ] pipeline = Pipeline(stages=indexer)

transformed = pipeline.fit(nd).transform(nd)

transformed.show()

String to index conversion

6. Creating training and test data

(training,test)=transformed.randomSplit([0.8, 0.2])

7. Creating ALS model and fitting data

als=ALS(maxIter=5,regParam=0.09,rank=25,userCol="reviewerID_index",itemCol="asin_index",ratingCol="overall",coldStartStrategy="drop",nonnegative=True) model=als.fit(training)

8. Generate predictions and evaluate rmse

evaluator=RegressionEvaluator(metricName="rmse",labelCol="overall",predictionCol="prediction") predictions=model.transform(test)

rmse=evaluator.evaluate(predictions) print("RMSE="+str(rmse))

predictions.show()

RMSE=1.168435412679992

Predictions for test data

9. Providing Recommendations

user_recs=model.recommendForAllUsers(20).show(10)

Recommendations

10. Converting back to string form

As seen in above image the results are in integer form we need to convert it back to its original name.The code is little bit longer given so many conversions.

import pandas as pd

recs=model.recommendForAllUsers(10).toPandas()

nrecs=recs.recommendations.apply(pd.Series) \

.merge(recs, right_index = True, left_index = True) \

.drop(["recommendations"], axis = 1) \

.melt(id_vars = ['reviewerID_index'], value_name = "recommendation") \

.drop("variable", axis = 1) \

.dropna()

nrecs=nrecs.sort_values('reviewerID_index')

nrecs=pd.concat([nrecs['recommendation'].apply(pd.Series), nrecs['reviewerID_index']], axis = 1)

nrecs.columns = [



'ProductID_index',

'Rating',

'UserID_index'



]

md=transformed.select(transformed['reviewerID'],transformed['reviewerID_index'],transformed['asin'],transformed['asin_index'])

md=md.toPandas()

dict1 =dict(zip(md['reviewerID_index'],md['reviewerID']))

dict2=dict(zip(md['asin_index'],md['asin']))

nrecs['reviewerID']=nrecs['UserID_index'].map(dict1)

nrecs['asin']=nrecs['ProductID_index'].map(dict2)

nrecs=nrecs.sort_values('reviewerID')

nrecs.reset_index(drop=True, inplace=True)

new=nrecs[['reviewerID','asin','Rating']]

new['recommendations'] = list(zip(new.asin, new.Rating))

res=new[['reviewerID','recommendations']]

res_new=res['recommendations'].groupby([res.reviewerID]).apply(list).reset_index() print(res_new)

Final Result

Voila!!

You just made an recommendation application.