Data Scraping and Analysis using Python
By: Jasmeet Singh
Tags: Towards Data Science, Data Scraping, Exploratory Data Analysis, Data Visualization, Retail

Data Scraping and Analysis using Python

Data Scraping is a technique to retrieve large amounts of data from the internet. This technique is highly useful in competitive pricing. To check what our product’s optimal price should be we can compare the similar products that are already in the market. These prices can vary a lot. So, in this blog, I’m going to show how we can scrap data regarding a particular product.

The most common technique for Data Scraping is using BeautifulSoup. It extracts the html for the page and stores it as an unstructured data. We’ll have to convert that into structured format.

Let’s import all the necessary libraries that we’ll require:

import requests

from fake_useragent import UserAgent

import pandas as pd

import bs4

The data that we extract is unstructured data. So we’ll create empty lists to store them in a structured form,

products=[] #List to store name of the product

prices=[] #List to store price of the product

ratings=[] #List to store rating of the product

specifications = [] #List to store specifications of the product

df=pd.DataFrame()

Creating a user agent. Refer to this link https://pypi.org/project/fake-useragent/

user_agent = UserAgent()

Taking the product name as an input. The extracted data will be related to that product.

product_name = input("Product Name- ")

To extract data from multiple pages of the product listing we’re going to use a for loop. The range will specify the number of pages to be extracted.



url = " for i in range(1,11):url = " https://www.flipkart.com/search?q={0}&page={1 }" url = url.format(product_name,i) ## getting the reponse from the page using get method of requests module

page = requests.get(url, headers={"user-agent": user_agent.chrome}) ## storing the content of the page in a variable

html = page.content ## creating BeautifulSoup object

page_soup = bs4.BeautifulSoup(html, "html.parser") for containers in page_soup.findAll('div',{'class':'_3liAhj'}):

name=containers.find('a', attrs={'class':'_2cLu-l'})

price=containers.find('div', attrs={'class':'_1vC4OE'})

rating=containers.find('div', attrs={'class':'hGSR34'})

specification = containers.find('div', attrs {'class':'_1rcHFq'})

products.append(name.text)

prices.append(price.text)

specifications.append(specification.text) if type(specification) == bs4.element.Tag else specifications.append('NaN')

ratings.append(rating.text) if type(rating) == bs4.element.Tag else ratings.append('NaN')

df = pd.DataFrame({'Product Name':products,'Price':prices, 'specification':specifications, 'Rating':ratings})

For extracting data form soup you need to specify the html tags you want to retrieve the data from. You could use inspect element on the webpage.

The above code will store the data in a structured format. And when you print the df you'll get:

df.head()

Cleaning up the data

Since all the products are the same we can write them as ‘bottle’.

df['Product Name'] = 'bottle'

df.head()

Similarly, we could find data for different products.

Now, we’ll remove the symbols from Price and clean up the specification column.

df[‘Price’] = df[‘Price’].str.lstrip(‘₹’)

df[‘Price’] = df[‘Price’].replace({‘,’:’’}, regex=True)

df.head()

df[‘Pack’], df[‘color’] = df[‘specification’].str.split(‘,’, 1).str

del df[‘specification’]

df.head()

Some basic Analysis of the data we got

Plotting a Boxplot

import numpy as np

import seaborn as sns

df[‘Price’] = df[‘Price’].astype(np.float)

sns.boxplot(x=df[‘Price’])

As we can see there are a few outliers where the price range is very high.

Plotting a bar plot

sns.barplot(x=df[‘Price’], y=df[‘color’])

We can observe how the prices for a particular colour changes. The rows with multiple colours are in packs like pack of 4 or pack of 6 etc.

df[‘Rating’] = df[‘Rating’].astype(np.float)

sns.barplot(x=df[‘Rating’], y=df[‘color’])

We can also observe that the colour has almost no effect on the ratings of the product.

sns.barplot(x=df[‘Rating’], y=df[‘Price’])

sns.lineplot(x=df[‘Rating’], y=df[‘Price’])

We can conclude from here that products with lower price have a higher ratings to some extent.

Thank you guys, I hope you got some insights on how we can use data scraping for competitive pricing. You could experiment with different EDA techniques other than mentioned in the Blog.