Web scraping with Selenium
By: X. Zhang
Tags: Selenium, Data Science, Web Scraping

Goal

After failing to find a public free dataset online, I finally decided to scrape my own data from dice.com for my project. The dataset should contain these rows: searched job title, similar job titles to the searched job title, skills related to the searched job title, the actual job title of each job postings, and the skill requirements of the job postings.

Inspecting the website

The link contains the search job title like this

Let’s say that we want to find all the job postings for research assistants. The searched job title is our own input, so we don’t need to scrape it from the website again. The input job title can also be easily changed in the URL itself (as well as the number of job postings per page), which enables us to search for different job titles later (and also to change the number of job postings per page).

The highlighted area contains data we want to scrape

When we open the link in Google Chrome, we can easily find the developer’s tool with Ctrl+Shift+I. However it’s very difficult to locate the data we want from the Elements by expanding each html tag and reading its content. Luckily enough when I tried to search the keyword “similar” with Ctrl+F, I was able to locate the similar titles immediately. And as you can see from the screenshot: the related skills are directly under it!

Writing the codes

As always we need to import our libraries first:

from selenium import webdriver import pandas as pd

Then we need to install the latest Chrome driver (an executable file) and put it in the same directory. With the driver object we can let selenium access the website.

driver = webdriver.Chrome(‘chromedriver’) driver.get(‘https://www.dice.com/jobs/q-research+assistant-limit-30-jobs’)

Getting the XPath

Then we can make use of the driver’s function find_element_by_xpath(‘XPath’) to get the specific web elements. The XPath can be easily copied from the developer’s tool page by right-clicking on the HTML element. This function returns a WebElement object, that has the function get_attribute(‘attribute_name’):

titles_element = driver.find_element_by_xpath(‘//*[@id=”similarSearchesTitle”]’) similar_titles = titles_element.get_attribute(‘value’) skills_element = driver.find_element_by_xpath(‘//*[@id=”similarSearchesSkills”]’) similar_skills = skills_element.get_attribute(‘value’)

When we print the similar_skills and similar_titles (which are Strings), we can see the data we want to extract printed. However, we’ll want to save it as a csv file when we do the same for 300 different job titles. This is when the pandas library comes to use:

data = pd.DataFrame(columns = [‘search_title’,’similar_titles’, ‘similar_skills’]) data.loc[n_titles] = [search_title,similar_titles,similar_skills]

Where n_titles is an int that is initially 0 and is incremented by 1 each time we search for a different job title on dice.com. And later we can write into a csv file using the function to_csv(‘file_name.csv’, index=False). Index can also be True if you want the rows to be numbered.

Go to each job postings

Now we have the basic information of each searched job title, how should we access individual job postings?

Link of the job posting is in the attribute href

If we inspect the website closely, we can find out that each job posting has an individual ID from 0 to the maximum number of job postings per page -1. Let’s iterate through all the position ids and collect all the links we will have to visit:

for position in range(0, 30): try: link_element = driver.find_element_by_xpath(‘//*[@id=”position’+str(position)+’”]’) except: continue links.append(link_element.get_attribute(‘href’))

Here I’ve included a try and except block, because it is also possible that the search result contains less than 30 job postings, and without a try block this would throw a No Such Element Exception.

Then we can have another loop going through each link in the list and doing the same to scrape the specific data of each job posting. For me it was the job title and the skill requirements:

for link in links: driver.get(link) try: job_title_element = driver.find_element_by_xpath(‘//*[@id=”estJTitle”]’) except: continue job_title = job_title_element.get_attribute(‘value’) skills = driver.find_element_by_xpath(‘//*[@id=”estSkillText”]’) required_skills = skills.get_attribute(‘value’) data.loc[n_jobs] = [search_title, job_title, required_skills] n_jobs+=1

Where n_jobs was initialized with 0. Here I’ve also used a try and except block, in case a job posting is expired.

Ending

So that was it, my first web scraping session :) It was surprisingly simple and straight-forward, and also much faster and convenient than using web-scraping softwares (which are mostly not free). For this particular project I’ve also scraped 300 IT job titles from another website which I searched on dice.com, and the procedure is exactly the same.

In the future, I’d like to explore web scraping with the more classical libraries such as Beautiful Soup and Scrapy (although the installation of the latter failed).