A Real-Time Product Recommender in Python | by Abhijit Roy | Towards Data Science

We probably can agree, those of us who are familiar with E-Shopping, have experience of shopping at Amazon. Over the years it has undoubtedly become one of the largest E-Shopping hubs. So, when we search for something, we can rely on Amazon’s Search Algorithms to provide us with several best choices, available based on our search terms. Let’s talk about how can we easily use a real-time product recommender based on Amazon’s Results and fine-tune our shopping experience.
The driving idea is to see how robust and user-friendly can we make our shopping procedure. Say, we want to buy a pair of headphones. If we search on Amazon, it will give a list of almost 25–30 products. Now, every buyer has a few preferences. The most significant of these preferences are brand Specification and pricing. Apart from these, buyers look at several other factors like Product popularity, Product Rating, Product Review, and Best Price match. Alongside, we as users also sometimes feel, it would have been great if we can at a glance see, what are the specifications several products are offering and make a decision from an expert point of view.
Now, needs vary from shopper to shopper. Say, some shoppers focus on reviews, some on ratings while others stick to price. So, won't it be great if we can provide the user with the power to decide, what he/she wants to focus on? So, let’s see how we can achieve these aims.
The idea is to assign each product, a score based on the fields, a user may look at during shopping. For example, we assign a product, a score based on popularity, a score based on reviews, and so on. Later, we calculate a weighted score based on user preferences.
Let’s see the idea with a concrete example. Say, we want to buy a headphone. So, we search and obtain a list of 25 items. We assign a variable score x1 to each product based on their ratings, x2 based on popularity, x3 based on reviews, and x4 based on price constraints. Now, we ask the user if he/she has any preferences like focussing more on some aspect than others. For instance, if the user wants to focus more on reviews we calculate the allover score as:
y= x1+x2+ax3+x4
This gives more weightage to x3. If the user has no such preferences then, we can calculate as.
y= x1+x2+x3+x4
Here we will standardize every value to a max of 1, to keep an equal balance of weightage of every factor. Next, we can sort the products based on the y value to get our results.
At the same time, we will create a specifications list of all the products to help our users with specific requirements to get an intuition about which product suits them the best.
Let’s jump to the application
If we search on amazon’s site our options appear as shown below.
We will scrap the options given by amazon’s search algorithm. We will do this using Selenium Webdriver.
If we look closely in the search bar and break it we can easily frame our direct search URL, which will help us land on the amazon search page.
import refrom selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.chrome.options import Optionsfrom bs4 import BeautifulSoupimport requestsdef search_am(phrase):    link="https://www.amazon.in/s?k="    l_end="&ref=nb_sb_noss"    phrase_w= phrase.replace(' ','+')    link_full=link+phrase_w+l_end    #print(link_full)        driver = webdriver.Chrome()        wait = WebDriverWait(driver, 5)    driver.get(link_full)        names_f=[]    names=driver.find_elements_by_tag_name("a")    i=0    for name in names:        className = name.get_attribute('class')        if className=='a-link-normal a-text-normal':            names_f.append(name)            i+=1                           links=[]    for i in names_f:        temp= i.get_attribute('href')        links.append(temp)            driver.quit()    return links
The above function helps us to scrape all the links of all the listed products on the search page, and return the links.
Each link leads us to a particular product page.
Next, we will focus on some parts of the product page.
The Rating Section:
The Rating Section:
This portion mentions the Rating of the product.
2. The Popularity Section: Here, I have used the number of ratings as a measure of popularity.
3. The Price Section: This portion gives the price of the product.
4. The Specifications Section: It lists all the product specifications and details.
5. The Review Section: This section reflects the reviews for the product.
Now, if we focus here in bold font a statement is there for each review. This line gives a gist idea about the review. We will pick up these sentences, judge the sentiment, and assign the review score.
Let’s first scrap the required details from the product pages.
def get_element_dets(link):    driver = webdriver.Chrome()    wait = WebDriverWait(driver, 2)    driver.get(link)    title_o= driver.find_elements_by_id("productTitle")    title=title_o[0].text        number_o= driver.find_elements_by_id("acrCustomerReviewText")    try:        popularity=(number_o[0].text)    except:        popularity='0'    rate=driver.find_elements_by_css_selector("#reviewsMedley > div > div.a-fixed-left-grid-col.a-col-left > div.a-section.a-spacing-none.a-spacing-top-mini.cr-widget-ACR > div.a-fixed-left-grid.AverageCustomerReviews.a-spacing-small > div > div.a-fixed-left-grid-col.aok-align-center.a-col-right > div > span > span")    try:        rate_o=(rate[0].text).split(' ')[0]    except:        rate_o='0'    feat_f=[]    tag=[]    value=[]    #features=driver.find_elements_by_css_selector("#feature-bullets > ul > li > span")    #for f in features:    #    feat_f.append(f.text)    price=0    try:        tag_o=driver.find_elements_by_tag_name('th')        for name in tag_o:            className = name.get_attribute('class')            if className=='a-color-secondary a-size-base prodDetSectionEntry':                tag.append(name.text)            value_o=driver.find_elements_by_tag_name('td')                for name in value_o:            className = name.get_attribute('class')            if className=='a-size-base':                value.append(name.text)        i=0        while i<len(value):            t=str(tag[i])+':'+str(value[i])            feat_f.append(t)            i+=1    except:        feat_f=[':']     try:        price_o= driver.find_elements_by_id("priceblock_ourprice")        for name in price_o:            className = name.get_attribute('class')            if className=='a-size-medium a-color-price priceBlockBuyingPriceString':                price=(name.text)                break    except:        price=0    #price=price_o.text        feedbacks=driver.find_elements_by_tag_name("a")        feedback_f=[]    for feed in feedbacks:        className = feed.get_attribute('class')        if className=='a-size-base a-link-normal review-title a-color-base review-title-content a-text-bold':            feedback_f.append(feed.text)                driver.quit()    return feedback_f,title,rate_o,popularity,feat_f,price
The above code snippet helps to scrap all details required from the product pages and return the product title feedback, rate, popularity, and other required values.
def caller(phrase):    links=search_am(phrase)    data={}    print(len(links))    for link in links:        data[link]={}        feedback_f,title,rate,popularity,feat_f,price=get_element_dets(link)        data[link]['feedback']=feedback_f        data[link]['title']=title        data[link]['rate']=rate        data[link]['popularity']=popularity        data[link]['features']=feat_f        if isinstance(price, int):            data[link]['price']=price                    else:                        data[link]['price']=price.split(' ')[1]        #print(len(data))    return data
The above snippet helps to arrange all the products and their corresponding features in a dictionary format.
The key to every product is its link, and then correspondingly all the features in a nested dictionary as key-value pairs.
Rarely Amazon pages have some variation in the tags, still, it's better to use a try and exception blocks to handle errors, just in case.
Now, after we have scrapped all the required data, let’s move to assign the scores.
def assign_popularity_rating():    with open('products.json', 'r') as openfile:              data = json.load(openfile)     temp=0    for k in data.keys():        p=int(data[k]['popularity'].split(' ')[0])        r=float(data[k]['rate'])        if p<50:            temp=1        elif p<100:            temp=2        elif p<150:            temp=3        else:            temp=4        score=(temp)        data[k]['Popularity_Score']=score        data[k]['Rating_Score']=r        with open("products_mod.json", "w") as outfile:             json.dump(data, outfile)
The above code is used to assign each product score based on a population and rating. For the population, I have used a binning or class approach. The ratings we already have from the values we obtained from scraping.
from textblob import TextBlobdef assign_sentiment_rating():    with open('products_mod.json', 'r') as openfile:              data = json.load(openfile)             sm=0    for k in data.keys():        temp=data[k]['feedback']              z=0        sm=0        for i in temp:            #print(i)            z+=1            t=TextBlob(i).sentiment.polarity            #print(t)            sm+=t        if (z==0):            rating=0        else:                        rating=sm/z        data[k]['Review_Score']=rating        with open("products_mod_2.json", "w") as outfile:             json.dump(data, outfile)
For review sentiment polarity detection, I have used TextBlob libraries’ sentiment polarity function. It assigns a value from -1 to +1 based on the sentiment detected from reviews. For a product, we have multiple reviews, so, for each review, we have a value. So, we add all the values obtained from all reviews and divide it by the number of reviews, in an attempt to keep the total score less than or equal to -1. Thus, we repeat the process for every product and obtain the rating for each.
def check_price_relevence():    with open('products_mod_2.json', 'r') as openfile:              data = json.load(openfile)     print("Specify the approx price to tune search")    price=int(input())    print("Specify a margin")    margin=int(input())        for k in data.keys():        data_ref=str(data[k]['price']).replace(',','')        temp=float(data_ref)                if temp<price+margin and temp>price-margin:            rating=1        else:            rating=0                    data[k]['Price_relevence_Score']=rating        with open("products_mod_3.json", "w") as outfile:             json.dump(data, outfile)
This is our function for price relevance. It asks for an approximate price and a margin for comparison. It then compares the price of the products and the range to assign the relevance score.
After the assignment of all scores, our dictionary adds the following for every product.
Next, we move to create a csv or excel file with the specifications of all listed products, for our customers with some specific needs.
import pandas as pddef form_featureset():    with open('products_mod_3.json', 'r') as openfile:                   data = json.load(openfile)        feat=[]        set_c=[]        for k in data.keys():            temp=data[k]['features']                        temp2=[]                    for i in temp:                tag=i.split(':')[0]                if tag not in feat:                    feat.append(tag)        #print(feat)        for k in data.keys():            temp=data[k]['features']            temp2=[-1]*len(feat)            for i in temp:                tag=i.split(':')[0]                                #print(tag)                                ind= feat.index(tag)                #print(ind)                                                    temp2[ind]= i.split(':')[1]                                            set_c.append(temp2)                        df=pd.DataFrame(set_c,columns=feat)    df.to_csv('product_descriptions.csv',index=False)    return df
This snippet produces a data frame with all products and their listed specifications to provide insight into the available specs.
The produced specification tables look like these. I have filled -1 for the values which were not given in the product pages.
These tables will be of help for a comparative search for the specifications, to the customers.
def tune_search(choice):with open('products_mod_3.json', 'r') as openfile:                   data = json.load(openfile)    for k in data.keys():            price_rel=data[k]['Price_relevence_Score']            review_score=data[k]['Review_Score']            pop_score=data[k]['Popularity_Score']            pop_score_k=pop_score/4                        rate_score=data[k]['Rating_Score']            rate_score_k=rate_score/5                        if choice==1:                total_score=5*pop_score_k+rate_score_k+review_score+price_rel            if choice==2:                total_score=pop_score_k+5*rate_score_k+review_score+price_rel            if choice==3:                total_score=pop_score_k+rate_score_k+review_score+5*price_rel            if choice==4:                total_score=pop_score_k+rate_score_k+5*review_score+price_rel                            else:                total_score=pop_score_k+rate_score_k+review_score+price_rel                            data[k]['Total_score']=total_score            #print(data[k]['Total_score'])    links=sort_d(data)                return links
This code snippet provides a score based on the choice of the user. I have used a very basic condition driven approach. I have divided the ratings by 5 and population by 4 to keep the values between 0 and 1. Here the weight value is fixed to 5. It’s just a random choice.
This is all about our code.
The given video demonstrates the application
Whatever the choice maybe, I have given the links for other choices too, just to give extra comfort and a chance to try other options to the user.
In my case, you will see the chrome window, though it is automated and closes on its own, keeps popping, you can stop it from launching, using headless chrome and availing it using the chromeoptions() for your chrome driver.
The application can be modified or made more robust in two ways, but both require data, which is currently not available, it seems(as far as I know).
If the current amazon sentiment datasets are available, we can make our own sentiment classifier where we can put other classes alongside just positive and negative sentiments, it will help our review based score to be more robust.If the data or specs are present categorically, we can create our own feature embedding for the specifications. For example, if we have enough laptop data instances, we can create an embedding or an encoding space for laptops. We can represent each laptop as an embedded vector of its specifications. We can create a new specification vector according to the user requirements. There, we can apply the K-Nearest Neighbours algorithm to obtain the K Nearest embedding vectors to the requirement vector and arrange them in order according to their euclidian distances. Thus we can obtain k laptops with specifications close to the user requirements. This will let us add a relevance of specifications based score, to make our system more robust.
If the current amazon sentiment datasets are available, we can make our own sentiment classifier where we can put other classes alongside just positive and negative sentiments, it will help our review based score to be more robust.
If the data or specs are present categorically, we can create our own feature embedding for the specifications. For example, if we have enough laptop data instances, we can create an embedding or an encoding space for laptops. We can represent each laptop as an embedded vector of its specifications. We can create a new specification vector according to the user requirements. There, we can apply the K-Nearest Neighbours algorithm to obtain the K Nearest embedding vectors to the requirement vector and arrange them in order according to their euclidian distances. Thus we can obtain k laptops with specifications close to the user requirements. This will let us add a relevance of specifications based score, to make our system more robust.
We can thus build a realtime product recommendation engine in python in a few steps.
Here is the Github Link.
Hope this helps.