Web Scraping with Selenium. This is the third part of a 4 part... | by Karthikeyan P | Towards Data Science

Selenium is a portable framework for testing web applications. It is open-source software released under the Apache License 2.0 that runs on Windows, Linux and macOS. Despite serving its major purpose, Selenium is also used as a web scraping tool. Without delving into the components of Selenium, we shall focus on a single component that is useful for web scraping, WebDriver. Selenium WebDriver provides us with an ability to control a web browser through a programming interface to create and execute test cases.
In our case, we shall be using it for scraping data from websites. Selenium comes in handy when websites display content dynamically i.e. use JavaScripts to render content. Even though Scrapy is a powerful web scraping framework, it becomes useless with these dynamic websites. My goal for this tutorial is to make you familiarize with Selenium and carry out some basic web scraping using it.
Let us start by installing selenium and a webdriver. WebDrivers support 7 Programming Languages: Python, Java, C#, Ruby, PHP, .Net and Perl. The examples in this manual are with Python language. There are tutorials available on the internet with other languages.
This is the third part of a 4 part tutorial series on web scraping using Scrapy and Selenium. The other parts can be found at
Part 1: Web scraping with Scrapy: Theoretical Understanding
Part 2: Web scraping with Scrapy: Practical Understanding
Part 4: Web scraping with Selenium & Scrapy
Installing Selenium on any Linux OS is easy. Just execute the following command in a terminal and Selenium would be installed automatically.
pip install selenium
Selenium officially has WebDrivers for 5 Web Browsers. Here, we shall see the installation of WebDriver for two of the most widely used browsers: Chrome and Firefox.
First, we need to download the latest stable version of chromedriver from Chrome’s official site. It would be a zip file. All we need to do is extract it and put it in the executable path.
wget https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_linux64.zipunzip chromedriver_linux64.zipsudo mv chromedriver /usr/local/bin/
Installing geckodriver for Firefox is even simpler since it is maintained by Firefox itself. All we need to do is execute the following line in a terminal and you are ready to play around with selenium and geckodriver.
sudo apt install firefox-geckodriver
There are two examples with increasing levels of complexity. First one would be a simpler webpage opening and typing into textboxes and pressing key(s). This example is to showcase how a webpage can be controlled through Selenium using a program. The second one would be a more complex web scraping example involving mouse scrolling, mouse button clicks and navigating to other pages. The goal here is to make you feel confident to start web scraping with Selenium.
Let us try out a simple automation task using Selenium and chromedriver as our training wheel exercise. For this, we would try to log into a Facebook account and we are not performing any kind of data scraping. I am assuming that you have some knowledge of identifying HTML tags used in a webpage using the browser’s developer tools. The following is a piece of python code that opens up a new Chrome browser, opens the Facebook main page, enters a username, password and clicks Login button.
from selenium import webdriverfrom selenium.webdriver.common.keys import Keysuser_name = "Your E-mail"password = "Your Password"# Creating a chromedriver instancedriver = webdriver.Chrome()  # For Chrome# driver = webdriver.Firefox() # For Firefox# Opening facebook homepagedriver.get("https://www.facebook.com")# Identifying email and password textboxesemail = driver.find_element_by_id("email")passwd = driver.find_element_by_id("pass")# Sending user_name and password to corresponding textboxesemail.send_keys(user_name)passwd.send_keys(password)# Sending a signal that RETURN key has been pressedpasswd.send_keys(Keys.RETURN)# driver.quit()
After executing this python code, your Facebook homepage would open in a new Chrome browser window. Let us examine how this became possible.
It all starts with the creation of a webdriver instance for your browser. As I am using Chrome, I have used driver = webdriver.Chrome().Then we open the Facebook webpage using driver.get("https://www.facebook.com"). When python encounters driver.get(URL), it opens a new browser window and opens the webpage specified by the URL.Once the homepage is loaded, we identify the textboxes to type e-mail and password using their HTML tag’s id attribute. This is done using driver.find_element_by_id().We send the username and password values for logging into Facebook using send_keys().We then simulate the user’s action of pressing RETURN/ENTER key by sending its corresponding signal using send_keys(Keys.RETURN).
It all starts with the creation of a webdriver instance for your browser. As I am using Chrome, I have used driver = webdriver.Chrome().
Then we open the Facebook webpage using driver.get("https://www.facebook.com"). When python encounters driver.get(URL), it opens a new browser window and opens the webpage specified by the URL.
Once the homepage is loaded, we identify the textboxes to type e-mail and password using their HTML tag’s id attribute. This is done using driver.find_element_by_id().
We send the username and password values for logging into Facebook using send_keys().
We then simulate the user’s action of pressing RETURN/ENTER key by sending its corresponding signal using send_keys(Keys.RETURN).
IMPORTANT NOTE:Any instance created in a program should be closed at the end of the program or after its purpose is served. So, whenever we are creating a webdriver instance, it has to be terminated using driver.quit(). If we do not terminate the opened instances, it starts to use up RAM, which may impact the machine's performance and slow it down. In the above example, this termination process has been commented out to show the output in a browser window. And, if terminated, the browser window would also be closed and the reader would not be able to see the output.
This is a more complex example. OpenAQ is a non-profit organization that collects and shares air quality data that are open and can be accessed in many ways. This is evident from the site’s robots.txt.
User-agent: * Disallow:
Our goal here is to collect data on PM2.5 readings from all the countries listed on http://openaq.org. PM2.5 are the particulate matter (PM) that have a diameter lesser than 2.5 micrometres, which is way smaller than the diameter of a human hair. If the reader is interested in knowing more about PM2.5, please follow this link.
The reason for choosing Selenium over Scrapy is that http://openaq.org uses React JS to render data. If it were static webpages, Scrapy would scrape the data efficiently. To scrape data, we first need to analyze the website, manually navigate the pages and note down the user interaction steps required to extract data.
It is always better to scrape with as few webpage navigations as possible. The website has a webpage https://openaq.org/#/locations which could be used as a starting point for scraping.
The filter locations option on the left-side panel is used to filter out PM2.5 data for each country. The Results on the right-side panel show cards that open a new page when clicked to display PM2.5 and other data.
A sample page containing PM2.5 data is shown below. From this page, we can extract PM2.5 values, location, city, country, date and time of recording PM2.5 value using XPATH or CSS.
Similarly, the left-side panel can be used to filter out and collect URLs of all the locations that contain PM2.5 data. The following are the actions that we performed manually to collect the data.
Open https://openaq.org/#/locationsFrom the left-side panel, select/click checkbox of a country. Let us go through the countries alphabetically.Also, from the left-side panel, select/click checkbox PM2.5.Wait for the cards to load in the right-side panel. Each card would then open a new webpage when clicked to display PM2.5 and other data.
Open https://openaq.org/#/locations
From the left-side panel, select/click checkbox of a country. Let us go through the countries alphabetically.
Also, from the left-side panel, select/click checkbox PM2.5.
Wait for the cards to load in the right-side panel. Each card would then open a new webpage when clicked to display PM2.5 and other data.
Based on the manual steps performed, data collection from http://openaq.org is broken down to 3 steps.
Collecting country names as displayed on OpenAQ countries webpage. This would be used in selecting appropriate checkboxes while filtering.Collecting URLs that contain PM2.5 data from each country. Some countries contain more than 20 PM2.5 readings collected from various locations. It would require further manipulation of the webpage, which is explained in the code section.Opening up webpages of the individual URL and extracting PM2.5 data.
Collecting country names as displayed on OpenAQ countries webpage. This would be used in selecting appropriate checkboxes while filtering.
Collecting URLs that contain PM2.5 data from each country. Some countries contain more than 20 PM2.5 readings collected from various locations. It would require further manipulation of the webpage, which is explained in the code section.
Opening up webpages of the individual URL and extracting PM2.5 data.
Now that we have the steps needed, let us start to code. The example is divided into 3 functions, each performing the task corresponding to the aforementioned 3 steps. The python code for this example can be found in my GitHub repository.
Instead of using OpenAQ locations webpage, there is https://openaq.org/#/countries webpage, which displays all the countries at once. It is easier to extract country names from this page.
from selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECimport jsondef get_countries():countries_list = []# driver = webdriver.Chrome() # To open a new browser window and navigate it# Use the headless option to avoid opening a new browser window    options = webdriver.ChromeOptions()    options.add_argument("headless")    desired_capabilities = options.to_capabilities()    driver = webdriver.Chrome(desired_capabilities=desired_capabilities)# Getting webpage with the list of countriesdriver.get("https://openaq.org/#/countries")# Implicit wait    driver.implicitly_wait(10)# Explicit wait    wait = WebDriverWait(driver, 5)    wait.until(EC.presence_of_element_located((By.CLASS_NAME, "card__title")))        countries = driver.find_elements_by_class_name("card__title")    for country in countries:        countries_list.append(country.text)driver.quit()# Write countries_list to json file    with open("countries_list.json", "w") as f:        json.dump(countries_list, f)
Let us understand how the code works. As always, the first step is to instantiate the webdriver. Here, instead of opening a new browser window, the webdriver is instantiated as a headless one. This way, a new browser window will not be opened and the burden on RAM would be reduced. The second step is to open the webpage containing the list of countries. The concept of wait is used in the above code.
Implicit Wait: When created, is alive until the WebDriver object dies. And is common for all operations. It instructs the webdriver to wait for a certain amount of time before elements load on the webpage.
Explicit Wait: Intelligent waits that are confined to a particular web element, in this case, tag with class name “card__title”. It is generally used along with ExpectedConditions.
The third step is to extract the country names using the tag with class name “card__title”. Finally, the country names are written to a JSON file for persistence. Below is a glimpse of the JSON file.
countries_list.json["Afghanistan", "Algeria", "Andorra", "Antigua and Barbuda", ... ]
The next step after getting the list of countries is to get the URLs of every location that records PM2.5 data. To do this, we need to open the OpenAQ locations webpage and make use of the left-side panel to filter out countries and PM2.5 data. Once it is filtered, the right-side panel would be populated with cards to individual locations that record PM2.5 data. We extract the URLs corresponding to each of these cards and eventually write them to a file that would be used in the next step of extracting PM2.5 data. Some countries have more than 20 locations that record PM2.5 data. For example, Australia has 162 locations, Belgium has 69 locations, China has 1602 locations. For these countries, the right-side panel on locations webpage is subdivided into pages. It is highly imperative that we navigate through these pages and collect URLs of all the locations. The code below has a while TRUE: loop that performs this exact task of page navigation.
from selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.common.action_chains import ActionChainsfrom logzero import loggerimport selenium.common.exceptions as exceptionimport timeimport jsondef get_urls():# Load the countries list written by get_countries()    with open("countries_list.json", "r") as f:        countries_list = json.load(f)        # driver = webdriver.Chrome()        # Use headless option to not open a new browser window    options = webdriver.ChromeOptions()    options.add_argument("headless")    desired_capabilities = options.to_capabilities()    driver = webdriver.Chrome(desired_capabilities=desired_capabilities)urls_final = []    for country in countries_list:# Opening locations webpage        driver.get("https://openaq.org/#/locations")        driver.implicitly_wait(5)        urls = []# Scrolling down the country filter till the country is visible        action = ActionChains(driver)        action.move_to_element(driver.find_element_by_xpath("//span[contains(text()," + '"' + country + '"' + ")]"))        action.perform()# Identifying country and PM2.5 checkboxes        country_button = driver.find_element_by_xpath("//label[contains(@for," + '"' + country + '"' + ")]")        values_button = driver.find_element_by_xpath("//span[contains(text(),'PM2.5')]")                # Clicking the checkboxes        country_button.click()        time.sleep(2)        values_button.click()        time.sleep(2)while True:            # Navigating subpages where there are more PM2.5 data. For example, Australia has 162 PM2.5 readings from 162 different locations that are spread across 11 subpages.locations = driver.find_elements_by_xpath("//h1[@class='card__title']/a")for loc in locations:                link = loc.get_attribute("href")                urls.append(link)try:                next_button = driver.find_element_by_xpath("//li[@class='next']")                next_button.click()            except exception.NoSuchElementException:                logger.debug(f"Last page reached for {country}")                breaklogger.info(f"{country} has {len(urls)} PM2.5 URLs")        urls_final.extend(urls)logger.info(f"Total PM2.5 URLs: {len(urls_final)}")    driver.quit()# Write the URLs to a file    with open("urls.json", "w") as f:        json.dump(urls_final, f)
It is always a good practice to log the output of programs that tend to run longer than 5 minutes. For this purpose, the above code makes use of logzero. The output JSON file containing the URLs looks like this.
urls.json[    "https://openaq.org/#/location/US%20Diplomatic%20Post%3A%20Kabul",    "https://openaq.org/#/location/Kabul",    "https://openaq.org/#/location/US%20Diplomatic%20Post%3A%20Algiers",    ...]
The process of getting PM2.5 data from the individual location is a straight forward web scraping task of identifying the HTML tag containing the data and extracting it with text processing. The same happens in the code provided below. The code extracts the country, city, location, PM2.5 value, URL of the location, date and time of recording PM2.5 value. Since there are over 5000 URLs to be opened, there would be a problem with RAM usage unless the RAM installed is over 64GB. To make this program to run on machines with minimum 8GB of RAM, the webdriver is terminated and re-instantiated every 200 URLs.
from selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.common.action_chains import ActionChainsfrom logzero import loggerimport selenium.common.exceptions as exceptionimport timeimport jsondef get_pm_data():# Load the URLs list written by get_urls()     with open("urls.json", "r") as f:        urls = json.load(f)# Use headless option to not open a new browser window    options = webdriver.ChromeOptions()    options.add_argument("headless")    desired_capabilities = options.to_capabilities()    driver = webdriver.Chrome(desired_capabilities=desired_capabilities)list_data_dict = []    count = 0for i, url in enumerate(urls):        data_dict = {}# Open the webpage corresponding to each URL        driver.get(url)        driver.implicitly_wait(10)        time.sleep(2)try:            # Extract Location and City            loc = driver.find_element_by_xpath("//h1[@class='inpage__title']").text.split("\n")            logger.info(f"loc: {loc}")            location = loc[0]            city_country = loc[1].replace("in ", "", 1).split(",")            city = city_country[0]            country = city_country[1]            data_dict["country"] = country            data_dict["city"] = city            data_dict["location"] = locationpm = driver.find_element_by_xpath("//dt[text()='PM2.5']/following-sibling::dd[1]").textif pm is not None:                # Extract PM2.5 value, Date and Time of recording                split = pm.split("μg/m3")                pm = split[0]                date_time = split[1].replace("at ", "").split(" ")                date_pm = date_time[1]                time_pm = date_time[2]                data_dict["pm25"] = pm                data_dict["url"] = url                data_dict["date"] = date_pm                data_dict["time"] = time_pmlist_data_dict.append(data_dict)                count += 1except exception.NoSuchElementException:            # Logging the info of locations that do not have PM2.5 data for manual checking            logger.error(f"{location} in {city},{country} does not have PM2.5")# Terminating and re-instantiating webdriver every 200 URL to reduce the load on RAM        if (i != 0) and (i % 200 == 0):            driver.quit()            driver = webdriver.Chrome(desired_capabilities=desired_capabilities)            logger.info("Chromedriver restarted")# Write the extracted data into a JSON file    with open("openaq_data.json", "w") as f:        json.dump(list_data_dict, f)logger.info(f"Scraped {count} PM2.5 readings.")    driver.quit()
The outcome of the program looks as shown below. The program has extracted PM2.5 values from 4114 individual locations. Imagine opening these individual webpages and manually extracting the data. It is times like this makes us appreciate the use of web scraping programs or bots, in general.
openaq_data.json[    {        "country": " Afghanistan",        "city": "Kabul",        "location": "US Diplomatic Post: Kabul",        "pm25": "33",        "url": "https://openaq.org/#/location/US%20Diplomatic%20Post%3A%20Kabul",        "date": "2020/07/31",        "time": "11:00"    },    {        "country": " Algeria",        "city": "Algiers",        "location": "US Diplomatic Post: Algiers",        "pm25": "31",        "url": "https://openaq.org/#/location/US%20Diplomatic%20Post%3A%20Algiers",        "date": "2020/07/31",        "time": "08:30"    },    {        "country": " Australia",        "city": "Adelaide",        "location": "CBD",        "pm25": "9",        "url": "https://openaq.org/#/location/CBD",        "date": "2020/07/31",        "time": "11:00"    },    ...]
I hope this tutorial has given you the confidence to start web scraping with Selenium. The complete code of the example is available in my GitHub repository. In the next tutorial, I shall show you how to integrate Selenium with Scrapy.
Till then, Good Luck. Stay safe and happy learning.!