Web Scraping JavaScript Content in Python with Selenium and BeautifulSoup
By: Neeraj Khadagade
Tags: Python, Web Scraping, JavaScript, Selenium, Programming

In the following article, you will see how easy, and seamless Selenium is when it comes to handling the JavaScript tags in an HTML files.

Let’s continue with the scraping.

In Step 4, we had stored the links of the companies in a list. We will be using these links to redirect to each company’s profile and extract information.

The list “links”

Step 5: Iterating through each company’s link in the list links.

First, we import the relevant libraries. This is where we start using Selenium.

Also, we create 2 items here:

A dictionary “data” to save the name, address, and the survey scores An empty dataframe “df”, where we would append and save the dictionary, data

# Importing the Selenium libraries

from selenium import webdriver

from selenium.webdriver import Chrome

from selenium.webdriver.chrome.options import Options data = {}

df = pd.DataFrame(columns = ['Name','Address'])

And we loop through each link from the list, links.

for link in links:

# Extract the scores

options = Options()

options.add_argument('--headless')

options.add_argument('--no-sandbox')

path = "C:\\Users\\lenovo\\Documents\\chromedriver.exe"

driver = webdriver.Chrome(path, options = options)

driver.get(link)

The last line driver.get(link) is where the Chrome Driver starts accessing each link.

NOTE: You will have to download the “chromedriver.exe” from this link if you don’t have it downloaded already. Make sure the version is the same as your current Google Chrome’s version.

Step 6: We are at the most interesting step, where we start fetching the scores. This is where we write our JavaScript query.

To extract the scores, we first create an empty list metric_values; we will fetch the list of dictionaries, chart.data, and append it. Since this list is under for loop, a new empty list will be created for each iteration of the link.

# Fetching the score from the JavaScript Tag

metric_values = []

# This is the JavaScript query to extract chart.data

results_vals = driver.execute_script

("""""

let results_vals = chart.data.map(({result})=>result);

console.log(results_vals);

return results_vals

""""") metric_values.append(results_vals)

We will use this metric_values to fetch the survey scores.

Step 7: Extract the name, address for each company, and save the scores for each survey scores.

# Extracting info

soup = requests.get(link)

webpage = BeautifulSoup(soup.content, 'html.parser') # Name of the company

data['Name'] = webpage.find('div', class_ = 'letterhead').find('h1', class_ = 'styled').get_text() try:

data['Address'] = webpage.find('div', class_ = 'contact-box').find('div', class_ = 'address contact').text.rstrip()

except:

data['Address'] = "Not Found" # Satisfaction Scores

try:

data['Satisfaction'] = ''.join([score[0].strip() for score in

metric_values])

except:

data['Satisfaction'] = "Not Found" # Expectations Scores

try:

data['Expectations'] = ''.join([score[1].strip() for score in

metric_values])

except:

data['Expectations'] = "Not Found" ... and so on for all the survey scores # Converting the list dictionary data into a dataframe

scores = scores.append(data, ignore_index=True)

At the end, we will convert the dictionary data to a dataframe.

The field “name” will act as the common key to merge both the dataframes profile (discussed in step 4 of Part 1), and scores.

Step 8: Merge both the dataframes

Finally, it’s time to merge both the dataframes on “Name”. And there you go!