Web Scraping Job Postings from Indeed.com using Selenium | by Bonnie Ma | Towards Data Science

I wish I knew how to web scraping earlier in my data science journey! It is such a handy tool and opens so many doors to cool projects. It may sound a little overwhelming to people who are new to coding. But trust me, it is easier and more intuitive than you thought!
This post is the Part I of a bigger project I recent accomplished which aimed to predict job salary for data related fields (data science, business intelligence, analytics, etc.) through text classification and deep learning. To build the model, I need to gather a large amount of job postings and the salary associated with the job posting.
Therefore, I decided to scrape jobs from Indeed.com. The API I am using is called Selenium. Selenium is widely used for web automation and it is actually very easy to install and use. You can install selenium like this:
pip install selenium
Selenium requires a driver to interface with the web browser. You can find Chrome, Edge, Firefox or Safari drivers here. After downloading, make sure the driver is in the same path with your browser app.
from selenium.webdriver.common.by import Byfrom selenium import webdriverfrom selenium.webdriver.common.keys import Keysfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as EC#specify driver pathDRIVER_PATH = '/Users/bonniema/Desktop/chromedriver'driver = webdriver.Chrome(executable_path = DRIVER_PATH)
The driver.get method will navigate to a page given by the URL. After executing this, you will see a webpage opening indicating that the web browser is being controlled.
driver.get('https://indeed.com')
Next step is to perform some job search. One advantage of using Selenium is that you can identify an item or button by id, name or xpath. In this case, I want to click the “Find jobs” button. If you inspect the page, you can see the HTML of “Find jobs” button. As highlighted in the picture below, we can tell the button element is in a <button> tag which is under a divider <div> that is under the form with id=”whatWhereFormId”. While you can have different ways to identify the button, I am going to use that form ID to identify the button. It is better to use an item or attribute (eg. class) that is unique enough, however, doesn’t change over time.
initial_search_button = driver.find_element_by_xpath(‘//*[@id=”whatWhereFormId”]/div[3]/button’)initial_search_button.click()
A pop up window shows after clicking the search button. We can close it down using
close_popup = driver.find_element_by_id("popover-close-link")close_popup.click()
After playing around with the page a little bit, I know that I want to perform Advanced Job Search where I can specify search terms and set numbers of jobs that get displayed per page. Looking at the HTML structure, looks like “Advanced Job Search” is wrapped in a <a> tag. One way is to use “contains” to locate the xpath by text.
advanced_search = driver.find_element_by_xpath("//a[contains(text(),'Advanced Job Search')]")advanced_search.click()
The other way is to right click that path and check if we can copy the XPath directly.
Seems this XPath does not his general enough to use here.
//*[@id="jobsearch"]/table/tbody/tr/td[4]/div/a
Next we need to send values to the search form. Below piece of code send position keywords, set display limit, and sort results by date
#search data science search_job = driver.find_element_by_xpath('//input[@id="as_and"]')search_job.send_keys(['data science'])#set display limit of 30 results per pagedisplay_limit = driver.find_element_by_xpath('//select[@id="limit"]//option[@value="30"]')display_limit.click()#sort by datesort_option = driver.find_element_by_xpath('//select[@id="sort"]//option[@value="date"]')sort_option.click()search_button = driver.find_element_by_xpath('//*[@id="fj"]')search_button.click()
Our goal is to get the position, company name, city, and company rating, salary (if there is any), and the job url from one job card, and then iterate through all the job cards on that page. Then go to the next page to scrape more data.
Below piece of code use a for loop to iterate through job cards in each page and save relevant information in a list.
Now the heavy lifting part has done!
descriptions=[]for link in links:        driver.get(link)    jd = driver.find_element_by_xpath('//div[@id="jobDescriptionText"]').text    descriptions.append(jd)
Put them all in a data frame!
df_da=pd.DataFrame()df_da['Title']=titlesdf_da['Company']=companiesdf_da['Location']="Palo Alto, CA"df_da['Link']=linksdf_da['Review']=reviewsdf_da['Salary']=salariesdf_da['Description']=descriptions