A beginner guide for scraping data from IMDB for User Reviews using Selenium and BeautifulSoup
By: Hung Pham
Tags: Selenium, Beautifulsoup, Data Scraping, Python, IMDb

A beginner guide for scraping data from IMDB for User Reviews using Selenium and BeautifulSoup Hung Pham Apr 13·7 min read

An ELI5 version for beginners to use Selenium and BeautifulSoup for data scraping.

Recently I had a project that required user review data from IMDB but couldn’t find any guide on the web for it. So I decided to learn Selenium and BS4 to implement them in this small project.

First thing first, let’s quickly review how Selenium and BS4 work and how are they different, as both are common tools to scrape data on the web for data analytic.

For this guide, I will assume that you (just like me) have no prior knowledge about HTML code but familiar with basic python functinalities.

The way that both use to scrap data is to load the whole webpage in its HTML form, then we can search and extract elements from the webpage using their HTML tag.

That being said, there are a few key differences between these two (Selenium has many other applications, so we will only talk about web scraping aspect here):

Selenium:

Selenium’s main application is web automation, which means we can program it to replicate what we want to do with a webpage, such as:

Open it (.get())

Find something within the web page (.find_element_by…())

Click on something (.click())

Close it (.quit())

Etc…We can do much more with Selenium, but that is out of the scope of this article.

BeautifulSoup:

BeautifulSoup(BS4) on the other hand, required the help from `requests` library to load the page, and only then extract the elements from the webpage.

BS4 is faster compared with selenium when extracting data.

So back to our main topic, our goal here is to extract user reviews of the top 50 highest rating movies. We can get these lists easily using google.

For this project, we will use this link:

The process, step-by-step would be:

Use Selenium to open the top 100 pages, and grab the link to each individual movie (We only need 50 so the first page would be enough).

Go to each movie’s page and grab the link to their respective `User Reviews` page.

Then we can go to each movie's ‘User Reviews’ page and extract the content of the reviews, store them into a list and create our dataframe.

All set, let’s check out some code:

** Note that I will put a link to github for the full notebook at the end, the embedded code here is for illustration purpose only **

1. Install the packages:

Both can be installed with pip in the command line:

pip install selenium pip install beautifulsoup4

2. Import the libraries:

import selenium #webdriver is our tool to interact with the webpage

from selenium import webdriver import requests #needed to load the page for BS4 from bs4 import BeautifulSoup import pandas as pd #Using panda to create our dataframe

3. Setup webdriver for Selenium and load the webpage

To use selenium, we will need to download the webdriver file corresponding with our browser. We’re using Chrome here: https://chromedriver.chromium.org/downloads

Make sure that our Chrome version is the same with the webdriver (for example, we will use the 89.0 version here, just the first 2 digit of the version is enough).

#path to the webdriver file

PATH = r"C:\chromedriver.exe" #tell selenium to use Chrome and find the webdriver file in this location

driver = webdriver.Chrome(PATH)

url = '

driver.get(url) #Set the url link and load the webpageurl = ' https://www.imdb.com/search/title/?groups=top_100' driver.get(url)

So from now on, we can treat the driver object here as the webpage itself and do all sorts of stuff as discussed above to it. If you run the code up until this point, you will see the website pop up.

4. Extract data

One of the most common tasks we want to do when scraping data is to locate and extract a specific element from the webpage. We can manually find it by inspecting the HTML code of the page.