Building a Product Recommendation Engine on Google Cloud’s Platform
Tags: Spark, Machine Learning, Recommender Systems, Cloud Computing, Google Cloud Platform

Building a Product Recommendation Engine on Google Cloud’s Platform

Image courtesy: Google

In my last post, I discussed how you could easily create an end to end recommendation engine using Amazon Sagemaker.

Today we will leverage the Google Cloud Platform and Apache Spark to create a recommendation engine that you can easily integrate with the data engineering pipeline. You can use the same mechanism and deploy it on AI platform notebooks.

Prerequisites:

You need a Google Cloud Platform subscription link. Access to google cloud services: cloud storage, dataproc, cloud SQL. Knowledge of PySpark and SQL.

Before we dive deep, let’s discuss a few terms.

Matrix Factorization:

It’s a collaborative filtering technique where user embedding and item embedding matrices are generated. The matrix product of both the matrix gives a close approximation of the user feedback for product recommendation. These matrix embeddings are learned over a period of time.

For more information, you can check my last post or check Google developer documentation.

Mathematically it can be represented using the following formula:

Where U and V are user embedding and item embedding matrix.

A is the dot product you want to represent close to the actual representation of user-item interaction. The distance is also known as the Frobenius distance.

In the real world, we also need to consider unobserved pairs into consideration, e.g., say a user is new and has never shown interest in a certain type of product than an existing user who has shown interest in more categories products.

Hence the objective function takes the below form, also called weighted matrix factorization.

Where w0 is the hyperparameter, which is tuned and is important to balance out the sum over observed and unobserved entities.

There are 2 algorithms to minimize the above objective functions:

SGD(stochastic gradient descent): generalized technique

pros: flexible and can be parallelized

cons: slower to converge, handling unobserved entries is difficult.

WALS(Weighted Alternating Least Squares): specialized for our objective.

Pros: fast to converge, parallelization, handles unobserved entries.

Cons: relies on loss square function.

Now let’s jump into the real-world problem.

Open the google cloud console following this link. Make sure to create or select a project for your work. Organization ->project ->folders is the folder hierarchy followed in the google cloud platform.

2. Open the navigation menu and create a Cloud SQL instance by navigating to database->SQL->create instance ->mySQL

Make sure to select a unique instance ID and root password.

3. Now, let’s create tables within MySQL.

Once the instance is ready, you can use google cloud shell to connect to the database.

At the top right, you will notice an icon to activate the cloud shell. Click the cloud shell.

Now let’s connect to our newly created database using a cloud shell.

gcloud sql connect rentals --user=root --quiet

If you are prompted to enable the SQL admin API, enable it following the link in error.

It might take up to 5 minutes to whitelist your IP.

4. Once you are in the MySQL console, create a database and 3 tables which will store the data for the recommendation.

CREATE DATABASE IF NOT EXISTS recommendation_spark; USE recommendation_spark; DROP TABLE IF EXISTS Recommendation; DROP TABLE IF EXISTS Rating; DROP TABLE IF EXISTS Accommodation; CREATE TABLE IF NOT EXISTS Accommodation ( id varchar(255), title varchar(255), location varchar(255), price int, rooms int, rating float, type varchar(255), PRIMARY KEY (ID) ); CREATE TABLE IF NOT EXISTS Rating ( userId varchar(255), accoId varchar(255), rating int, PRIMARY KEY(accoId, userId), FOREIGN KEY (accoId) REFERENCES Accommodation(id) ); CREATE TABLE IF NOT EXISTS Recommendation ( userId varchar(255), accoId varchar(255), prediction float, PRIMARY KEY(userId, accoId), FOREIGN KEY (accoId) REFERENCES Accommodation(id) ); SHOW DATABASES;

Right now, the tables will be empty. You will stage the data in cloud SQL and then pull it to the cloud SQL.

5. Create a new window in the cloud shell and create buckets and pull data into cloud storage buckets.

echo “Creating bucket: gs://$DEVSHELL_PROJECT_ID” gsutil mb gs://$DEVSHELL_PROJECT_ID echo “Copying data to our storage from public dataset” gsutil cp gs://cloud-training/bdml/v2.0/data/accommodation.csv gs://$DEVSHELL_PROJECT_ID gsutil cp gs://cloud-training/bdml/v2.0/data/rating.csv gs://$DEVSHELL_PROJECT_ID echo “Show the files in our bucket” gsutil ls gs://$DEVSHELL_PROJECT_ID echo “View some sample data” gsutil cat gs://$DEVSHELL_PROJECT_ID/accommodation.csv

You can check the content by going to the navigation menu -> storage -> browser -> project ID.

6. Let’s load the data from the cloud storage bucket to cloud SQL.

Navigate to SQL->rentals->import data

Specify the bucket, database, and table name and click import.

Repeat the above steps for rating data.

7. Explore the data using cloud shell and running a few commands

SELECT userId, COUNT(rating) AS num_ratings FROM Rating GROUP BY userId ORDER BY num_ratings DESC;

8. Let’s create a recommendation engine using dataproc.

Enable the dataproc API by navigating to the API and services->dashboard-> search Cloud Dataproc API.

Once done navigate to BigData-> Dataproc->Cluster->create cluster

Make sure to keep the zone the same as your cloud SQL instance.

Also, note the name, zone, and total worker nodes in your cluster.

8. Authorize dataproc to connect to the cloud SQL instance

Use the below script in the cloud shell. Make sure to check the cluster name, zone, and a number of workers.

echo “Authorizing Cloud Dataproc to connect with Cloud SQL” CLUSTER=cluster-a20f CLOUDSQL=rentals ZONE=us-central1-f NWORKERS=2 machines=”$CLUSTER-m” for w in `seq 0 $(($NWORKERS — 1))`; do machines=”$machines $CLUSTER-w-$w” done echo “Machines to authorize: $machines in $ZONE … finding their IP addresses” ips=”” for machine in $machines; do IP_ADDRESS=$(gcloud compute instances describe $machine — zone=$ZONE — format=’value(networkInterfaces.accessConfigs[].natIP)’ | sed “s/\[‘//g” | sed “s/’\]//g” )/32 echo “IP address of $machine is $IP_ADDRESS” if [ -z $ips ]; then ips=$IP_ADDRESS else ips=”$ips,$IP_ADDRESS” fi done echo “Authorizing [$ips] to access cloudsql=$CLOUDSQL” gcloud sql instances patch $CLOUDSQL — authorized-networks $ips

9. Copy the ML model to the datarproc cluster and edit it using cloud shell.

gsutil cp gs://cloud-training/bdml/v2.0/model/train_and_apply.py train_and_apply.py cloudshell edit train_and_apply.py

Below is the pyspark code snippet

import os import sys import pickle import itertools from math import sqrt from operator import add from os.path import join, isfile, dirname from pyspark import SparkContext, SparkConf, SQLContext from pyspark.mllib.recommendation import ALS, MatrixFactorizationModel, Rating from pyspark.sql.types import StructType, StructField, StringType, FloatType # MAKE EDITS HERE CLOUDSQL_INSTANCE_IP = ‘104.155.188.32’ # ← — CHANGE (database server IP) CLOUDSQL_DB_NAME = ‘recommendation_spark’ # ←- leave as-is CLOUDSQL_USER = ‘root’ # ←- leave as-is CLOUDSQL_PWD = ‘root’ # ← — CHANGE # DO NOT MAKE EDITS BELOW conf = SparkConf().setAppName(“train_model”) sc = SparkContext(conf=conf) sqlContext = SQLContext(sc) jdbcDriver = ‘com.mysql.jdbc.Driver’ jdbcUrl = ‘jdbc:mysql://%s:3306/%s?user=%s&password=%s’ % (CLOUDSQL_INSTANCE_IP, CLOUDSQL_DB_NAME, CLOUDSQL_USER, CLOUDSQL_PWD) # checkpointing helps prevent stack overflow errors sc.setCheckpointDir(‘checkpoint/’) # Read the ratings and accommodations data from Cloud SQL dfRates = sqlContext.read.format(‘jdbc’).options(driver=jdbcDriver, url=jdbcUrl, dbtable=’Rating’, useSSL=’false’).load() dfAccos = sqlContext.read.format(‘jdbc’).options(driver=jdbcDriver, url=jdbcUrl, dbtable=’Accommodation’, useSSL=’false’).load() print(“read …”) # train the model model = ALS.train(dfRates.rdd, 20, 20) # you could tune these numbers, but these are reasonable choices print(“trained …”) # use this model to predict what the user would rate accommodations that she has not rated allPredictions = None for USER_ID in range(0, 100): dfUserRatings = dfRates.filter(dfRates.userId == USER_ID).rdd.map(lambda r: r.accoId).collect() rddPotential = dfAccos.rdd.filter(lambda x: x[0] not in dfUserRatings) pairsPotential = rddPotential.map(lambda x: (USER_ID, x[0])) predictions = model.predictAll(pairsPotential).map(lambda p: (str(p[0]), str(p[1]), float(p[2]))) predictions = predictions.takeOrdered(5, key=lambda x: -x[2]) # top 5 print(“predicted for user={0}”.format(USER_ID)) if (allPredictions == None): allPredictions = predictions else: allPredictions.extend(predictions) # write them schema = StructType([StructField(“userId”, StringType(), True), StructField(“accoId”, StringType(), True), StructField(“prediction”, FloatType(), True)]) dfToSave = sqlContext.createDataFrame(allPredictions, schema) dfToSave.write.jdbc(url=jdbcUrl, table=’Recommendation’, mode=’overwrite’)

When prompted, click open in a new window.

Change the cloud instance IP and root password.

Now save and copy it to the bucket. We will use dataproc to pick the code from the storage location.

gsutil cp train_and_apply.py gs://$DEVSHELL_PROJECT_ID

10. Time to submit ML job to the dataproc cluster

Navigate to dataproc->job-> submit job

Specify job id, cluster, job type as pyspark, main python file bucket path, and click submit the job.

11. Query the recommendation table

Once the job completes successfully, you can query the MySQL table recommendation for a specific recommendation based on user_id.

Note: Make sure to delete resources and shut down services to avoid any surprises in the billing from google cloud.

Do let me know your thoughts. You can follow me for tutorials on AI/machine learning, data analytics, and BI. You can connect with me on LinkedIn.

References

_http://yifanhu.net/PUB/cf.pdf

_https://spark.apache.org/docs/latest/ml-collaborative-filtering.html

_http://stanford.edu/~rezab/classes/cme323/S15/notes/lec14.pdf

_https://developers.google.com/machine-learning/recommendation/collaborative/matrix