Step-by-Step Web Scraping Project Using Selenium in Python
By: Lasha Gochiashvili
Tags: Selenium, Selenium Webdriver, Web Scraping

The boxplot shows the data about PM 2.5 air pollution levels in EU countries. PM 2.5 is particulate matter that is measured in the air. In simple words — bigger the number worst the quality of air.

What will we automate with Selenium:

The manual process of collecting PM 2.5 data from the website

This article covers the following topics:

Selecting a website Setting up Selenium in Python Scraping data from the website & storing Analyzing data with Seaborn boxplot in Python

1. Selecting a website

Before you decide which website to scrap, here comes the natural question — is this even legal to scrap data from the website?

If you want to find out whether scraping is allowed on the website or not, first you should check the information on the link: add “robots.txt” at the end of the website. E.g: www.examplewebsite.com/Robots.txt

Let’s look if we can scrap Facebook.com. When you open the link www.Facebook.com/Robots.txt you will see the following information:

It clearly says that you need special authorization from Facebook to scrap the data. For a better understanding of Robots.txt please check more info here.

It is always recommended to use scraping on the websites that are licensed under creative commons, have an open data source, or do not prohibit scraping in the Robots.txt.

Due to the fact that this project is non-commercial and the website www.openaq.org has open data that can be accessed in many ways, we decided to use it for scraping. They even encourage people to use data for good in their awesome FAQ worth checking!

What data to collect?

On this website, we decided to scrap data about country & PM 2.5 information. To do that, we want to filter on PM 2.5 and for each country to access location cards. Then copy the PM 2.5 data (which is a number). Basically, we want to do the same as shown in the below gif.

The manual process of collecting PM 2.5 data from the website

It looks tiring and boring, right? Accessing pages of around 100 countries, each country has up to 15 locations. Wohooo!!! No way. But the good news is that Selenium can automate it for us. But! We need to set this up.

2. Setting up Selenium in Python

You should install Selenium in your computer. Simply typing pip install selenium in Command Prompt (for Windows) and hit the ENTER. For more details about installation please see Selenium documentation.

Installing Selenium from Command Prompt

Then you need a driver which will be a bridge for Selenium to access your browser. In my case, I downloaded a geckodriver for Mozilla Firefox. Depends for your version you can download it from here:

Firefox: https://github.com/mozilla/geckodriver/releases

Considering that you downloaded geckodriver, unzip it, and copy-paste geckodriver.exe into the folder where your python is.

Copy-paste geckodrive.exe into the folder where python app is located (my case)

Create a .py file and import main libraries and packages.

Main Libraries and Packages for Selenium

Then we need to find the path to our geckodriver.exe and set up webdriver in python code as shown below.

WebDriver Settings in Python

3. Scraping data from the website & storing

The scraping workflow looks like this:

Scraping 4-step workflow

3.1 Get the names of the countries that are in the filter

We need country names exactly with the same spelling as they are in the filter. Getting values from the filter seems a bit challenging. Instead, we can get all country names from this link. They will be with the same spelling.

If you right-click on the country name and go to Inspect Element you will see that country names are saved in class card__title.