How to Scrape Dynamic Web pages with Selenium and Beautiful Soup
By: Alan Jones
Tags: Beautifulsoup, Programming, Web Scraping, Hands On Tutorials, Python

How to Scrape Dynamic Web pages with Selenium and Beautiful Soup

Beautiful Soup is a great tool for extracting data from web pages but it works with the source code of the page. Dynamic sites need to be rendered as the web page that would be displayed in the browser — that’s where Selenium comes in. Alan Jones Nov 21, 2020·5 min read

Image by Author

Beautiful Soup is an excellent library for scraping data from the web but it doesn’t deal with dynamically created content. That’s not in any way a criticism — Beautiful Soup does precisely the job it is supposed to do and that does not include rendering the webpage as a browser would.

In order to get that dynamic content the web page must be interpreted by a browser so the Javascript that creates the dynamic content can do its work. But how do we get at the HTML code that is rendered by the browser? One answer is by using a headless browser and the Selenium Python library. And that is what I’m going to demonstrate here.

We are going to use a very simple HTML file that contains some dynamically rendered text. Here it is:

<html>

<head>

<title>Dynamic web page</title>

</head>

<body>

<div id=”text”>Placeholder text</div>

<script>

document.getElementById(“text”).innerHTML=”This is the text we want”

</script>

</body>

</html>

All we have here is a HTML file with a single <div> in the body that contains some text (“Placeholder text”) but when the page is loaded that text is replaced with the text generated by the Javascript further down. The script locates the <div> by its id, “text” and then sets the internal text to : “This is the text we want”.

So, if we view the page in a web browser we see the text that is dynamically generated by the Javascript.

The browser view — image by author

Let’s see what happens when we run this by Beautiful Soup and try and scrape the text from the <div>.

from bs4 import BeautifulSoup

import os soup_file=open(os.getcwd() +"\\test.html")

soup = BeautifulSoup(soup_file)

print(soup.title.get_text())

print(soup.find(id=’text’).get_text())

The code above imports BeautifulSoup and the os library, opens the file test.html from the local directory and creates a BS object which is stored in the variable soup.

Then we have two print statements. The first gets the text from the title in the BS object and prints that. The second does a similar thing but finds the tag with the id “text” and gets the text from that.

And the result?

Dynamic web page

Placeholder text

Oh dear — not what we want. BeautifulSoup is correctly parsing the code but finding the default text in the <div> rather than the text that was rendered in the browser. If we had been looking at a table of say stock market prices that were filled from data in a database, we would likely see only null values in the table, not the dynamically loaded prices.

What we need is the HTML to be run in a browser in order to see the correct values and then be able to capture those values programmatically.

The way to do this is with a headless browser. A headless browser is essentially a browser with no user interface. It does not render its output to a screen but rather to a text object.

Most, if not all, modern browsers will run in headless mode but they need a driver in order to communicate the results back to the user. Also, in order to use the results in a Python program, we need a library that will talk to the driver.

The Python library is Selenium and the driver that it communicates with is known as a webdriver. In the example below, I will use Chrome as the headless browser and so I need the Chrome web driver which you can download from chromedriver.chromium.org/downloads. If you use a different browser just search for “webdriver Edge”, “webdriver Firefox”, etc to find the appropriate download. Then download the webdriver and place it in your working directory. You will also need to conda/pip install Selenium.

The first thing to do is import the necessary parts of Selenium and then set the appropriate options. In the code below, the “ — headless” argument will tell the browser that it should run in headless mode (obviously), then I’ve specified the location of the Chrome binary. As you can see I’m using a portable version of Chrome and it is located in a folder in my working directory — I like to keep things self-contained. You may not need to specify where the binary is if you have a standard installation.

The webdriver is in the local directory and I set a variable to its path.

Next I instantiate a driver object using the previously set options and the location of the Chrome webdriver. With this driver I can load a web page which will be interpreted by the Chrome browser. The result will be loaded into the driver object where we can access the text of the page in the page_source attribute.

from selenium import webdriver

from selenium.webdriver.chrome.options import Options # Instantiate an Options object

# and add the “ — headless” argument

opts = Options()

opts.add_argument(“ — headless”) # If necessary set the path to you browser’s location

opts.binary_location= os.getcwd() +’\\GoogleChromePortable\GoogleChromePortable.exe’ # Set the location of the webdriver

chrome_driver = os.getcwd() +”\\chromedriver.exe” # Instantiate a webdriver

driver = webdriver.Chrome(options=opts, executable_path=chrome_driver) # Load the HTML page

driver.get(os.getcwd() +”\\test.html”)

The next step is to create a Beautiful Soup object and load the page source into it. We can then scrape data from this source. In the code below you can see that we do much the same as in the previous exercise. But this time the result will be different. Here’s the code:

# Put the page source into a variable and create a BS object from it

soup_file=driver.page_source

soup = BeautifulSoup(soup_file) # Load and print the title and the text of the <div>

print(soup.title.get_text())

print(soup.find(id=’text’).get_text())

And this is the result:

Dynamic web page

This is the text we want

As you can see, now we are using the code that has been processed by the headless web browser, the result is what would be rendered in a browser window, not the original source as in our first attempt.

Finally, we need to close the browser:

# Close the browser

driver.quit()

And that is all there is to it. With the code above and your own Beautiful Soup code, you are now equipped to start scraping data from dynamic web pages.