A Detailed Web Scraping Walkthrough Using Python and Selenium
By: Kenneth Leung
Tags: Selenium, Healthcare, Python, Webscrapping, Data

Section 3 — Running Main Script

With the same dependencies installed and imported earlier, we can proceed to build a script to automate the tedious web scrapping process. I previously mentioned that there are various healthcare professional bodies on the MOH webpage, and they are actually represented with different abbreviations. For example, SDC for Singapore Dental Council, SMC for Singapore Medical Council etc. For the Pharmacists group, the abbreviation is SPC.

We prepare our WebDriver once again to access the page and target HTML frame, along with several WebDriver options to make sure it runs smoothly. In particular, we are introducing wait times and sleep times so that there can be appropriate pauses during the scraping process.

(i) WebDriver Initiation

# Define healthcare professional (HCP) body

hcp_body = 'SPC'



# Set wait times

waittime = 20

sleeptime = 2



# Initiate web driver

try: # Close any existing WebDrivers

driver.close()

except Exception:

pass



# Access professional registration system (PRS) for specific

# healthcare professional body

home_page = f"https://prs.moh.gov.sg/prs/internet/profSearch/main.action?hpe={hcp_body}"



# Set webdriver options

options = webdriver.ChromeOptions()

options.add_argument('--no-sandbox')

options.add_argument('ignore-certificate-errors')



# Initiate webdriver

driver = webdriver.Chrome(options=options)



# Get driver to retrieve homepage

driver.get(home_page)



# Switch to frame which contains HTML for Search section

driver.switch_to.frame(driver.find_element_by_name('msg_main'))



# Click Search button (named btnSearch) to load all results

WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, "//input[@name='btnSearch']"))).click()



# Sleep a short while for page loading to be fully completed

time.sleep(sleeptime)

(ii) Setting up key functions

Because there is a series of actions to take in order to navigate through the different records across different search results pages, it is best to setup functions to better organize all these steps.

(1) Setup master list CSV to store all the records

We first generate a CSV file to store the records which we will be appending later. The column names are based on the earlier exploration of the web page, where we identified the various fields we can extract from each public record.

# Set file name (based on your own preference)

file_name = 'master_list.csv'



# Check if file already exists

if os.path.isfile(f'./{file_name}'):

print(f'Filename {file_name} already exists')

else:

# Set names of fields we want to extract

column_names = ['name','reg_number','reg_date','reg_end_date','reg_type','practice_status','cert_start_date','cert_end_date','qualification','practice_place_name','practice_place_address','practice_place_phone']



# Generate template dataframe

df_template = pd.DataFrame(columns = column_names)



# Generate csv file from the dataframe template

df_template.to_csv(f'{file_name}', header=True)



print('Created new master list file')

(2) Get current page number

We need a function to track which page of the Search results the WebDriver-controlled browser is currently on

def get_current_page(): # Wait until element is located, then click it

current_page_elem = WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, "//label[@class='pagination_selected_page']"))).text



# Get page number as integer

current_page_num = int(current_page_elem)



return current_page_num

(3) Get absolute last page number

It is important to know how many Search results pages we have in total, so that we can create a page loop accordingly. For example, at the time of this exercise, the last page number is 340 (i.e. there are 340 Search results pages)

def get_absolute_last_page(): # Find all elements with pagination class (since it contains

# page numbers)

WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, "//a[@class='pagination']")))

all_pages = driver.find_elements_by_xpath("//a[@class='pagination']")



# Get final element, which corresponds to 'Last' hyperlink

# (which will go to last page number)

last_elem = all_pages[-1].get_attribute('href')



# Keep only the number of last page

last_page_num = int(re.sub("[^0-9]", "", last_elem))



return last_page_num

(4) Extract data from detailed information page of a single record

While we are on the details page of a particular healthcare professional (after clicking the ‘View More Details’ link), we need to have a way to extract all the text from the respective fields available for public viewing. The information for each record is consolidated into a Python dictionary.

Do note that we are longer using BeautifulSoup here (since BS was used just for demo purposes earlier). We are now solely depending on Selenium to do the parsing and text extraction for us.

def gen_hcp_dict(): WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, "//div[@class='table-head']")))

hcp_name = driver.find_element_by_xpath("//div[@class='table-head']").text

all_fields = driver.find_elements_by_xpath("//td[@class='no-border table-data']") # Using find elementS since there are multiple elements

hcp_data = []



# Extract test for every field within the table

for field in all_fields:

hcp_data.append(field.text)



hcp_dict = {}



# Assign extracted data into respective key-value pairs

hcp_dict['name'] = hcp_name

hcp_dict['reg_number'] = hcp_data[0]

# hcp_data[1] is just a blank space, so it can be ignored

hcp_dict['reg_date'] = hcp_data[2]

hcp_dict['reg_end_date'] = hcp_data[3]

hcp_dict['reg_type'] = hcp_data[4]

hcp_dict['practice_status'] = hcp_data[5]

hcp_dict['cert_start_date'] = hcp_data[6]

hcp_dict['cert_end_date'] = hcp_data[7]

hcp_dict['qualification'] = hcp_data[8]

hcp_dict['practice_place_name'] = hcp_data[9]

hcp_dict['practice_place_address'] = hcp_data[10]

hcp_dict['practice_place_phone'] = hcp_data[11]



return hcp_dict

(5) Get current pagination range

There is a pagination range that is available on each page of the Search results, and it is usually limited to around 10 digits. This means that you are only able to directly access a particular page from that specific range visible to you. Thus, we need a function to determine what is the current pagination range that is accessible. Below is the screenshot of the pagination range I am referring to:

def get_current_pagination_range():



WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, "//a[@class='pagination']")))



# Get all elements related to pagination

all_pages = driver.find_elements_by_xpath("//a[@class='pagination']")



# Implicit wait to allow page to load

driver.implicitly_wait(1)



# Find numbers of pagination range, and append to list

pagination_range_on_page = []

for elem in all_pages:



# Only extracting numeric values of pagination range

if elem.text.isnumeric():

pagination_range_on_page.append(int(elem.text))

driver.implicitly_wait(1)

else:

pass

driver.implicitly_wait(1)

return pagination_range_on_page

(6) Click last pagination number on current page

With the pagination range known, we need a function to click on the last number displayed in this range. This is for us to proceed further into the pages beyond the given range. For example, to reveal pages beyond page 10, we need to click the number 10 from the range (Page 1 2 3 4 5 6 7 8 9 10 ..)

def click_last_pagination_num(pagination_range): last_pagination_num = pagination_range[-1]



# Introduce implicit wait for page to complete loading

driver.implicitly_wait(1)



# Click page number once element is located

WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, f'{last_pagination_num}'))).click()

driver.implicitly_wait(1)

(7) Click first pagination number on current page

Since there are hundreds of pages, it does not make sense to always start from page 1. This is because it will require an extremely tedious amount of effort to click on the last visible page number of each pagination range to reach the later half of the Search results.

The solution to this is that for page numbers greater than the halfway mark (e.g. pages beyond 100 if there is a total of 200 pages), our program will instead start from the last page, and go back in reverse to reach those latter pages. This strategy of starting from the last page to reach these latter pages is much more efficient compared to always starting from the first page.

To make that happen, we need a function to click the first number of the existing pagination range. For example, if our current pagination range is (Page .. 151 152 153 154 155 156 157 158 ..), and we want to go to page 147, the program will be clicking the page number 151 so as to reveal the preceding set of pagination range.

def click_first_pagination_num(pagination_range):



first_pagination_num = pagination_range[0]



# Introduce implicit wait for page to complete loading

driver.implicitly_wait(1)



# Click page number once element is located

WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, f'{first_pagination_num}'))).click()

driver.implicitly_wait(1)

(8) Locate and access target page

Let’s put the earlier functions into action. We now need to find a way to get our WebDriver to go to each page to scrape all the relevant information for each of the 10 professionals listed.

A massive pain comes from the fact that when you click the ‘Back to Search Results’ link while on the details page of a particular professional, it brings you all the way back to Page 1 of the Search results instead of the current page. This means that if you were looking at the details page of Mr Chan on Page 10 and you click ‘Back to Search Results’, you will be brought back to Page 1 instead of the page 10 you were on.

As such, we need a function that can repeatedly click the appropriate numbers of the pagination range to return to the target page we were scraping every time after the ‘Back to Search Results’ link is clicked. The following code sets up the automation to do just that, with the strategic use of conditional statements and loops.

def locate_target_page(target_page):



last_page_num = get_absolute_last_page()



# Find midway point of all search results page

midway_point = last_page_num/2



# If target page is in first half, start clicking from the start

if target_page < midway_point:

current_page_num = get_current_page()



if current_page_num == target_page:

pass

else:

pagination_range = get_current_pagination_range()



while target_page not in pagination_range:

driver.implicitly_wait(1)



# If target page is not in pagination range, keep

# clicking last pagination number to go down the list

click_last_pagination_num(pagination_range)

current_page_num = get_current_page()

pagination_range = get_current_pagination_range()

driver.implicitly_wait(1)

else:

WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, f"{target_page}"))).click() # Once target page is in pagination page, go to the target page # If target page is in later half of list, go to Last page and

# move in reverse (This saves alot of time)

else:



WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, 'Last'))).click()

time.sleep(sleeptime)

current_page_num = get_current_page()



if current_page_num == target_page:

pass

else:

pagination_range = get_current_pagination_range()



while target_page not in pagination_range:

driver.implicitly_wait(2)

click_first_pagination_num(pagination_range)

current_page_num = get_current_page()

pagination_range = get_current_pagination_range()

else:

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.LINK_TEXT, f"{target_page}"))).click()

# Once target page is in pagination page, go to the target page

(9) Create script that automates the navigation through portal

And now we setup the last function, which will bring everything together

def full_scrape(target_page):



last_page_num = get_absolute_last_page()

driver.implicitly_wait(1)



while target_page != last_page_num:

locate_target_page(target_page)

print('Starting with target page ' + str(target_page))



# Retrieve HTML from search page

target_page_html = driver.find_element_by_xpath("//body").get_attribute('outerHTML')

driver.implicitly_wait(1)



# Find list of all IDs on that page, and keep the unique IDs

all_ids = re.findall("P[0-9]{5}[A-Z]{1}", target_page_html)

id_list = list(dict.fromkeys(all_ids))



for index, hcp_id in enumerate(id_list): # Tracking the healthcare professional (HCP)'s ID

# Click 'View More Details' link to access details page

# for that professional with the specific ID

WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, f"//a[contains(@onclick,'{hcp_id}')]"))).click()



# Scrape data from details page into a dictionary

hcp_dict = gen_hcp_dict()



# Convert dict to pandas dataframe (Need to set an index

# since we are passing scalar values)

df_hcp_dict = pd.DataFrame(hcp_dict, index=[0])



# Append df row directly to existing master list csv

df_hcp_dict.to_csv(f'{file_name}', mode='a', header=False)



# Print row that was just scraped (To track progress)

print(f'Scraped row {index+1} of page {target_page}')



# Update (+1) target page after successfully scrapping

# all 10 records on page

if index == len(id_list):

print(f'Completed scraping for page {target_page}')

target_page += 1

print('Updated target page ' + str(target_page))

else:

pass



# Head back to homepage by clicking 'Back to Search

# Results' link

WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, 'Back to Search Results'))).click()



# Go to latest updated target page

locate_target_page(target_page)



# Define steps when program reaches last page (essentially

# similar to the steps above)

else:

locate_target_page(target_page)

print('Working on last page')

target_page_html = driver.find_element_by_xpath("//body").get_attribute('outerHTML')

driver.implicitly_wait(1)

all_ids = re.findall("P[0-9]{5}[A-Z]{1}", target_page_html)

id_list = list(dict.fromkeys(all_ids))



for index, hcp_id in enumerate(id_list):

WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, f"//a[contains(@onclick,'{hcp_id}')]"))).click()

hcp_dict = gen_hcp_dict()

df_hcp_dict = pd.DataFrame(hcp_dict, index=[0])

df_hcp_dict.to_csv('master_list.csv', mode='a', header=False)

print(f'Scraped row {index+1} of {target_page}')



if index == len(id_list)-1:

print(f'Completed scraping for page {target_page}')

print('Mission Complete')

else:

pass



WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, 'Back to Search Results'))).click()

locate_target_page(target_page)

(iii) Kickstart automated web scraping

It’s been a relatively intense process so far, in terms of setting up all the functions correctly. This will require plenty of trial and error, so do prepare to spend some time and effort to get the automation right. Putting things in perspective, this short term hassle will still deliver much much more efficiency compared to doing this scraping manually.

Now that we are done with the hardest part of the coding, so it is time to execute all our functions and witness the fruits of our labor.

# Start with selected target page. target_page = 1 by default (to start from page 1)

target_page = 1



# Display time started

print(time.strftime("%H:%M:%S", time.localtime()))



# Run main web scraping function

full_scrape(target_page)



# Display time concluded

print(time.strftime("%H:%M:%S", time.localtime()))

NOTE: You will notice many wait times peppered across the script. This is because the scraping will fail if a particular page is not loaded completely. Therefore, these wait times provide some buffer time for page loading. More info on waits can be found here.

Despite that, the webpage loading may still fail from time to time. If that is the case, simply take note which page was last scraped successfully (from the output screen in the above cell). You then update the target_page in the above chunk, and rerun the automated scraping (you may do so by re-running the code from the WebDriver initiation section).

(iv) Data Cleaning of Master List

Once scraping is completed, we can now tidy up the master list CSV file where our data has been progressively appended to. This is important since there will likely be row duplicates if you encounter scraping failures along the way and need to re-run scraping from break points.

# Import master list csv file

df_master = pd.read_csv(f'{file_name}') # Display master list columns

df_master.columns # Drop leftmost index column (Unnecessary column)

df_master.drop(columns = [df_master.columns[0]], axis = 1, inplace = True)



# Remove duplicates

df_master.drop_duplicates(keep='first', inplace=True) # Export final curated list

df_master.to_excel('master_list_final.xlsx', index = False)

(v) Conclusion

And that’s it! We have successfully performed web scrapping on a website that is relatively difficult to navigate. The concepts for the other types of websites are also similar, so you are in good stead to try out web scraping for other kinds of projects. If you do encounter specific errors or problems, always refer to trusty resources like StackOverflow and Selenium documentation pages.

Please feel free to reach out to me on LinkedIn should you have any suggestions or queries.

Happy web scraping!