The Ultimate Guide to Build a Competitor Price Monitor Dashboard | by Low Wei Hong | Towards Data Science

1
2
3
4
5
6
7
8
9
10
Powered by Play.ht
Create audio with Play.ht
Create Audio Narrations with Play.ht
Feeling that your work is full of repetitive work?
Why not start to think on how to automate it?
Here is the thing, today I am going to introduce a tool to automate your BORING stuff — Python. Python is perhaps the easiest language to learn. Because of your acquired Python skill, you will be able not only to increase your productivity but also focus on work which you will be more interested in.
Welcome to the Get Rid of boring stuff using Python series! Feel free to visit this link and you will be able to view all the stories in this ongoing series.
Now let's imagine you are a phone seller selling products on an e-commerce platform.
You will have a list of your competitors...
You would want to monitor how your competitor is setting their price so that you can slightly undercut your price to be more competitive.
Another scenario, imagine that you working as a salesperson in an e-commerce company and you are having a client, Adidas.
So, you will need to monitor whether Adidas is selling the same price across each e-commerce platform, if not then maybe you are able to question them on their pricing as well.
The price monitoring dashboard is extremely important, especially in e-commerce. When two sellers are selling the same product, price is the deciding factor of whether the customer will choose to purchase the product.
Let’s get started!
I will use an example of you being a seller in Amazon, selling Huawei P30 Pro. So, today you will be setting up a really simple dashboard — Google Sheet, to monitor how competitive your price is.
This is the agenda which I will be sharing:
Part 1 — Input details of your competitor.
Part 2 — Get your google sheet’s credential file.
Part 3 — Web Scraping on your competitor’s website [visualisation]
Part 4 — Web Scraping on your competitor’s website [coding]
Follow the whole journey and you will notice how simple it is to automate your boring stuff and to update your price IN 5 Seconds.
Create a google sheet and input the example competitor’s link. Set a price for your Huawei P30 Pro.
Define a conditional format for column Competitive, with the rule that if your price is lower than your competitor, it will be 1 and the cell will be filled with green color. Otherwise, set it to be 0 and fill the cell with red color as an alert that your competitor is setting a lower price than yours.
Follow this link, step by step, you will be able to retrieve your credential file in a JSON format. In the JSON format credential file, you will be able to see your email account as shown in the purple box below.
Then share your service account’s email to your google sheet which you have just created, as shown in the snapshot below.
Then click Send button, then you are done!
First, launch the competitor’s website in google chrome, right-click and choose Inspect, then you will be able to view the snapshot below in Snapshot 1.
Then, click the purple bracket on the top center, and you will have a cursor, then click the red bracket to inspect the HTML element of the price for the product as shown below (Snapshot 2).
Next, notice that the purple box above in Snapshot 2, that is the path we need to extract:
HTML tagname: span
HTML tagname <span> attribute <id>: priceblock_ourprice
*If you are not familiar with HTML, this will be a good reference for the above terminology.
So now let’s get started with the coding part!
Follow the above gist for connection to google sheet.
Do note that the input to the gc.open_by_key() is as follow: 11DU3qbPoXlbUXJ-o_zerndJdqQ8pNBMsAZpeBfbwyc0. This is actually the unique identifier for Google to find your corresponding google sheet.
You can get this unique identifier as shown below (purple box):
Retrieve data from the google sheet with sheet_name: Sheet1, by using get_all_values() function.
Then, store the value into the pandas data frame as shown below:
df = pd.DataFrame(sht1.worksheet("Sheet1").get_all_values()[1:])
YOU ARE 70% DONE!
Remember that HTML tag for price:
HTML tag <span> attribute <id>: priceblock_ourprice
So now, we use Beautiful Soup parser to get the element you need.
Think of soup format as below:
soup.find( HTML tag, { tag attribute: attribute value})
In our case:
HTML tagname: span
tag attribute: id
attribute value: priceblock_ourprice
soup = BeautifulSoup(res.text)price = soup.find("span",                  {"id": "priceblock_ourprice"}                 ).text.replace("$","")
Using update_acell(cell number, value) function to insert the price we scraped to the google sheet AND WE ARE DONE!
sht1.worksheet("Sheet1").update_acell('D3', price)
Combine everything mention above with the following code:
I am currently working as a Data Scientist. Data is growing really fast, so equip yourself with a good scraping skill is important as you will be able to obtain more data for your machine learning model.
Thank you for reading this post. Feel free to leave comments below on topics that you may be interested to know. I will be publishing more posts in the future about my experiences and projects.
Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipelines and also implementing machine learning models on solving business problems.
He provides crawling services that are able to provide you the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.
You can connect with him on LinkedIn and Medium.