Web Scraping Using Python and BeautifulSoup
By: Shivan Kumar
Tags: Image Scraping, Python, Web Scraping, Data Scientist, Data Science

Imagine you have a large amount of data from your websites and you want to do it as quickly as possible. How would you do it without manually going to each website and getting the data? Well, “Web Scraping” is the answer. Web Scraping just makes this job easier and faster.

In this article on Python with Web Scraping, you will learn about web scraping in brief and see how to extract data from a website with a demonstration.

what is a Web Scraper?

The Internet has a lot of data, and if you can get that data programmatically, you can analyze and visualize that data amazingly for your personal interest or even for research purposes. To ‘get’ this data, you need to make a web scraper, which is a program that can automatically extract data from a website.

The data on the websites are unstructured. Web scraping helps collect these unstructured data and store it in a structured form. There are many different ways to scrape the data such as online Services, through APIs, or writing your own custom code.

Why is Web Scraping Used?

Web scraping is used to collect large information from our websites. But why we collect such large data from websites? To know about this, let’s look at the applications of web scraping:

Social Media Scraping: Wen scraping is used to collect large data from any Social Media Platform such as Facebook, Instagram, Twitter to find out what’s trending topics is ongoing.

Price Comparison: Services such as Flipkart & Amazon website, we use this website to collect data from online shopping websites and use it to compare the prices of products.

Research and Development: Web scraping is used to collect a large set of data (Statistics, General Information, Temperature, etc.) from websites, which are analyzed and used to carry out Surveys or for R&D.

Job listings: Details regarding job openings, interviews are collected from different websites and then listed in one place so that it is easily accessible to the user.

There are many Popular Tools & Libraries used for Web Scraping in Python

As we know, Python has various applications and there are different libraries for different purposes. Here are three popular,

1. Selenium

2. BeautifulSoup

3. Scrapy

Let’s discuss one by one:

Selenium

Selenium is a popular tool for automating browsers. It’s primarily used for testing in the industry but is also very handy for web scraping.

BeautifulSoup

BeautifulSoup is an amazing parsing library in Python that enables web scraping from HTML and XML documents. It creates parse trees that are helpful to extract the data easily.

BeautifulSoup automatically detects encodings and gracefully handles HTML documents even with special characters. We can navigate a parsed document and find what we need which makes it quick and painless to extract the data from the webpages. In this article, we will learn how to build web scrapers using Beautiful Soup in detail.

Scrapy

Scrapy is a python framework for large scale web scraping. It gives you all the tools you need to efficiently extract data from websites, process them as you want, and store them in your preferred structure and format.

Web Scraping Example: Scraping nytimes Website

Pre-requisites:

Python 3.x with Selenium, BeautifulSoup, pandas libraries installed

Google-chrome browser

Let’s get started!

Step 1: Find the URL that you want to scrape

For this example, we are going scrape the nytimes website to extract the data where Trump has Lies (Trump’s Lies), The URL for this page is: https://www.nytimes.com/interactive/2017/06/23/opinion/trumps-lies.html

Step 2: Inspecting the Page

The data is usually nested in tags. So, we inspect the page to see, under which span/tags the data we want to scrape is nested. To inspect the page, just right click on the element and click on “Inspect”.

When you click on the “Inspect” tab, you will see a “Browser Inspector Box” open.

Step 3: Find the data you want to extract

Let’s extract the data where Trump has Lies (Trump’s Lies) which is in the “span” tag respectively.

Step 4: Write the code

First, let’s create a Python file. To do this, open the terminal in windows and type Jupyter notebook <your file name> with .ipynb extension.

First, let us import all the necessary libraries:

import requests from bs4 import BeautifulSoup

Refer to the below code to open the URL:

r = requests.get('https://www.nytimes.com/interactive/2017/06/23/opinion/trumps-lies.html')

Now that we have written the code to open the URL, it’s time to extract the data from the website. As mentioned earlier, the data we want to extract is nested in <span> tags. So, I will find the span tags with those respective class-names, extract the data, and store the data in a variable. Refer the code below:

soup = BeautifulSoup(r.text, 'html.parser') results = soup.find_all('span', attrs={'class':'short-desc'}) results

Now store your data in the form of a list:

records = []

for result in results:

date = result.find('strong').text[0:-1] + ', 2017'

lie = result.contents[1][1:-2]

explanation = result.find('a').text[1:-1]

url = result.find('a')['href']

records.append((date, lie, explanation, url))

Let’s check the first three data its store in the form of a list or not :

records[0:3]

Step 6: Store the data in a required format

After extracting the data, you might want to store it in a format. This format varies depending on your requirement. For this example, we will store the extracted data in a CSV (Comma Separated Value) format. To do this, I will add the following lines to my code:

import pandas as pd

df = pd.DataFrame(records, columns=['date', 'lie', 'explanation', 'url'])

df['date'] = pd.to_datetime(df['date'])

df.to_csv('trump_lies_datascet.csv', index=False, encoding='utf-8')

Now, I’ll run the whole code again.

A file name “trump_lies_datascet.csv” is created and this file contains the extracted data.

Conclusion

I hope you guys enjoyed this article on “Web Scraping Using Python and BeautifulSoup”. If you’re new to web scraping, the above examples can serve as a starting point for your future scraping adventures. All web pages are different, so the above scripts will naturally have to be modified for other pages, but the overall process should be the same.

Stay safe and happy Learning!