Data Scraping and Analysis using Python | by Jasmeet Singh | Towards Data Science

Data Scraping is a technique to retrieve large amounts of data from the internet. This technique is highly useful in competitive pricing. To check what our product’s optimal price should be we can compare the similar products that are already in the market. These prices can vary a lot. So, in this blog, I’m going to show how we can scrap data regarding a particular product.
The most common technique for Data Scraping is using BeautifulSoup. It extracts the html for the page and stores it as an unstructured data. We’ll have to convert that into structured format.
Let’s import all the necessary libraries that we’ll require:
import requestsfrom fake_useragent import UserAgentimport pandas as pdimport bs4
The data that we extract is unstructured data. So we’ll create empty lists to store them in a structured form,
products=[] #List to store name of the productprices=[] #List to store price of the productratings=[] #List to store rating of the productspecifications = [] #List to store specifications of the productdf=pd.DataFrame()
Creating a user agent. Refer to this link https://pypi.org/project/fake-useragent/
user_agent = UserAgent()
Taking the product name as an input. The extracted data will be related to that product.
product_name = input("Product Name- ")
To extract data from multiple pages of the product listing we’re going to use a for loop. The range will specify the number of pages to be extracted.
for i in range(1,11):    url = "https://www.flipkart.com/search?q={0}&page={1}"    url = url.format(product_name,i)    ## getting the reponse from the page using get method of requests module    page = requests.get(url, headers={"user-agent": user_agent.chrome})    ## storing the content of the page in a variable    html = page.content    ## creating BeautifulSoup object    page_soup = bs4.BeautifulSoup(html, "html.parser")    for containers in page_soup.findAll('div',{'class':'_3liAhj'}):        name=containers.find('a', attrs={'class':'_2cLu-l'})        price=containers.find('div', attrs={'class':'_1vC4OE'})        rating=containers.find('div', attrs={'class':'hGSR34'})        specification = containers.find('div', attrs {'class':'_1rcHFq'})        products.append(name.text)        prices.append(price.text)        specifications.append(specification.text) if type(specification) == bs4.element.Tag  else specifications.append('NaN')        ratings.append(rating.text) if type(rating) == bs4.element.Tag  else ratings.append('NaN')    df = pd.DataFrame({'Product Name':products,'Price':prices, 'specification':specifications, 'Rating':ratings})
For extracting data form soup you need to specify the html tags you want to retrieve the data from. You could use inspect element on the webpage.
The above code will store the data in a structured format. And when you print the df you'll get:
df.head()
Since all the products are the same we can write them as ‘bottle’.
df['Product Name'] = 'bottle'df.head()
Similarly, we could find data for different products.
Now, we’ll remove the symbols from Price and clean up the specification column.
df[‘Price’] = df[‘Price’].str.lstrip(‘₹’)df[‘Price’] = df[‘Price’].replace({‘,’:’’}, regex=True)df.head()
df[‘Pack’], df[‘color’] = df[‘specification’].str.split(‘,’, 1).strdel df[‘specification’]df.head()
import numpy as npimport seaborn as snsdf[‘Price’] = df[‘Price’].astype(np.float)sns.boxplot(x=df[‘Price’])
As we can see there are a few outliers where the price range is very high.
sns.barplot(x=df[‘Price’], y=df[‘color’])
We can observe how the prices for a particular colour changes. The rows with multiple colours are in packs like pack of 4 or pack of 6 etc.
df[‘Rating’] = df[‘Rating’].astype(np.float)sns.barplot(x=df[‘Rating’], y=df[‘color’])
We can also observe that the colour has almost no effect on the ratings of the product.
sns.barplot(x=df[‘Rating’], y=df[‘Price’])
sns.lineplot(x=df[‘Rating’], y=df[‘Price’])
We can conclude from here that products with lower price have a higher ratings to some extent.
Thank you guys, I hope you got some insights on how we can use data scraping for competitive pricing. You could experiment with different EDA techniques other than mentioned in the Blog.