How To Automate Web Scraping using BeautifulSoup for Dummies
By: Shan Yi Tan
Tags: Python, Automation, Data Science, Web Scraping, Data Visualization

1. Using Requests to download page

To start scraping a web page, first we need to download the page using the Python requests library . The requests library will make a GET request to a web server, which will download the HTML contents of a given web page for us. There are several different types of requests we can make using requests, of which GET is just one.

You can easily install the library by running code below

pip install requests

2. Using BeautifulSoup to parse the HTML content

To parse our HTML document and extract the 50 div containers, we’ll need to install a Python module called BeautifulSoup:

pip install BeautifulSoup4

Then, we will:

Import the BeautifulSoup class creator from the package bs4.

Parse response.text by creating a BeautifulSoup object, and assign this object to html_soup .

3. Understanding the HTML structure

Before you get all hyped up for web scraping, you need to understand the HTML of the website which you want to scrape from. Take note that every website has different structure.

Hover to the items you want to reach its HTML code

Right click on the website Left click on Inspect Turn on the hover cursor button on top left.

Each movie is in a div tag with class lister-item-mode-advanced . Let’s use the find_all() method to extract all the div containers that have a class attribute of lister-item mode-advanced :

As shown, there are 50 containers, meaning to say 50 movies listed on each page.

Now we’ll select only the first container, and extract, by turn, each item of interest:

The name of the movie.

The year of release.

The IMDB rating.

The Metascore.

Directors

The number of votes.

Gross

Let’s get started with the first_movie

4. Scraping Data

From the first_movie html which we had stored, we are going to use find and find_all with str slicing to work out the magic.

The name of the movie.

The year of release.

IMDB rating

Metascore

Directors

This is more complicated as this class contains Directors and Stars. So I used slicing and splitting to extract only the directors. You may use the same logic to extract Stars as well.

The number of votes

Gross

5. Changing the URL’s parameters (Where automation starts)

The URLs follow a certain logic as the web pages change. As we are making the requests, we’ll only have to vary the values of only two parameters of the URL:

release_date : Create a list called years_url and populate it with the strings corresponding to the years 2000-2017.

: Create a list called and populate it with the strings corresponding to the years 2000-2017. page : Create a list called pages, and populate it with the strings corresponding to the first 4 pages.

6. Controlling the crawl-rate

We need 2 functions:

sleep() : Control the loop’s rate. It will pause the execution of the loop for a specified amount of seconds. randint() : To mimic human behavior, we’ll vary the amount of waiting time between requests. It randomly generates integers within a specified interval.

7. Monitoring the loop as it’s still going

Monitoring is very helpful in the testing and debugging process, especially if you are going to scrape hundreds or thousands of web pages in a single code run. Here are the following parameters that we are gonna monitor:

The frequency (speed) of requests: make sure our program is not overloading the server.

Frequency value = the number of requests / the time elapsed since the first request.

The number of requests : can halt the loop in case the number of expected requests is exceeded.

: can halt the loop in case the number of expected requests is exceeded. The status code of our requests: make sure the server is sending back the proper responses.

Let’s experiment with this monitoring technique at a small scale first.

8. Piece everything together (Loops) 🎉

Phew~ Tough work is done, now let’s piece together everything we’ve done so far.

Import necessary libraries Re-declare the lists variables so they become empty again. Prepare the loop. Loop through the years_url list in the interval 2010–2019 and loop through the pages list in the interval 1–4. Make the GET requests within the pages loop Give the headers parameter the right value to make sure we get only English content. Pause the loop for a time interval between 8 and 15 seconds. Throw a warning for non-200 status codes. Break the loop if the number of requests is greater than expected. Convert the response‘s HTML content to a BeautifulSoup object. Extract all movie containers from this BeautifulSoup object. Loop through all these containers. Extract the data if a container has a Metascore. Extract the data if a container has a Gross, or else append(“-”).

9. Transform the scraped data into CSV

In the next code block we:

Merge the data into a pandas DataFrame.

Print some information about the newly created DataFrame.

Show the last 10 entries.

Sneak peak of the dataset

Last but not least, save the DataFrame to CSV, so that we can do data wrangling and EDA later on:

FREE access to all running code

Here’s the GitHub link to get the Python code. 👉

To be published next: Data Wrangling and EDA of movie ratings dataset

Stay tuned! 😊