Real World Example on Web Scraping with Selenium and Beautiful Soup | by Timothy Tan | Towards Data Science

5.39 Billion.
That’s how many pages the World Wide Web has got.
In terms of data?
Over 1,200 petabytes worth.
That’s equivalent to 1,200,000 terabytes, and 1,200,000,000 gigabytes.
All I’m saying is that there is a hell lot of unstructured data out there on the web.
Just sitting there...
Waiting to be scraped and analyzed. 😉
In this post, I will be going through code snippets on how to scrape the web with Selenium and Beautiful Soup.
I will start off with an introduction to the technologies before proceeding to an actual real world example of it.
I will also be surfacing issues I came across while working on this scraping project so you are made aware of them how to mitigate them for your own work.
Let’s get started!
For the uninitiated, let’s begin by a brief introduction of the technologies we will be using.
Born in 2004, Selenium is an open-sourced automated testing framework that is used to validate web applications across different browsers and platforms.
These test scripts can be built using different programming languages such as Java, C#, PHP, Python, Ruby and Perl.
In this post, we will be using a Selenium Python package (Selenium Web Driver) to create the test scripts.
As a quick introduction to how it works, here’s a GIF on me working with Selenium off a Jupyter Notebook.
In the GIF, I started off importing the relevant packages and initializing a chrome browser. At the same time, I told the browser to get “http://www.google.com”.
Next, I triggered a command to search “Selenium python API” in the Google Search bar before exiting.
If you would like to try it out, go ahead and start up your local Jupyter Notebook.
The first thing you will need to do is to install selenium.
pip install selenium
You will also need to install chrome driver here.
Then run the code below step by step.
Notice on line 13, the class name is “gLFyf.gsfi”?
How did I know that the Google Search Bar uses this tag?
To find out the different tags of a HTML page, all you need to do is to right-click on the page and enter developer mode with “inspect”.
Next, click on the icon at top left hand corner. This will enable you to select elements on page.
To view the tags for the search bar, simply hover your mouse over the search bar.
Once you find the relevant class tag you are interested in, you can tell Selenium to grab it with the code
elem = browser.find_element_by_class_name("gLFyf.gsfi")
A note of caution here though.
From experiences in this particular scraping project, I noticed a bug in the “find_element_by_class_name” function.
If the class names got too long with too many spaces, this function will fail to work.
I have Googled for tons of solutions. Even if you put those periods between the tokens i.e. “gLFyf<period>gsfi”
It doesn’t always work...
Very unreliable in this aspect.
A better way to filter out the class tags in particular is to use Beautiful Soup.
So if Selenium caused me such pain...
Why Selenium?
The reason why I had to use Selenium for this project was due to a limitation in Beautiful Soup.
When Beautiful Soup parses a HTML page, it will not parse the sections of the page that needs to be loaded by java script.
Sometimes, there are web pages that require the user to scroll down before a new section of a web page is loaded — the loading is usually done by java script.
Thus, if your web page require you to scroll down to load the segments of the page, the Beautiful Soup parser will never parse the full HTML.
To mitigate this issue, you should use Selenium to automate the scrolling process before parsing the HTML page in it’s entirety.
That way, you have a flexibility to either use Selenium or Beautiful Soup to start your information extraction process.
I used both because each of them had their own merits and together, they are like a power couple. 😃
Created in 2004, Beautiful Soup is a Python library made for parsing HTML and XML files seamlessly.
What do I mean by that?
Well, let’s go through a modified version of their example from their documentation.
I modified the example to make things a little clearer.
The first thing to do is to install Beautiful Soup.
pip install beautifulsoup4
Here are the codes:
As you can see, Beautiful Soup is pretty easy and intuitive to use.
I suggest you try them out yourself if you have the time.
Alright then!
Now that introductions are out of the way, here comes the meat of the post!
Let’s get to it!
If you went through the introductory steps above, well done!
But here’s where web scraping can get tad bit more complex.
The whole scraping project was to scrape post titles from TDS writers.
I wanted to analyze what topics TDS writers were writing about and approximate reader interest through the number of claps received per post.
I will not be going through the results in this post but it will be in another post.
However, I will be covering in the codes used to accomplish the scraping task.
This took me about 2 days to get it working as I faced a lot of different bugs along the way.
Here we go!
I accomplished this in two steps.
Step 1 — Define the web pages that needed to be scraped and find the common tags used through the differing pages.
This step takes awhile because you will need to find all relevant tags and cross-reference them with other pages to determine if they are pointing to the same element you want to scrape.
I did this manually by opening up multiple tabs of different pages and entering the developer mode i.e. right-click “Inspect” or hit F12.
Then I hovered on the elements I wanted to extract and searched for a common tag that all of these elements use.
Let’s take my profile for example.
The information of interest to me were:
Publisher Name
Date Posted
Read Time
Title of Post
No. of Claps
What I was looking for were the “div” tags that hold the information I want extracted.
Here, I found the “div” tag I needed and also noticed it had “class” tags!
Perfect!
Notice how there are 7 identical “class” tags above?
That’s because each post in my profile page has the same tag name.
<div class = "gk gl gm gn go y c">...</div>
This is something you would want to look out for.
But the question is, do other user profiles follow the same tag?
Let’s cross-reference this finding with another profile.
It turns out, not quite.
<div class = "gm gn go gp gq y c">...</div>
But after taking a look at another profile...
There is a pattern.
<div class = "go gp gp gr gs y c">...</div>
The pattern to all the “div” tags that I need follows a regex pattern of:
re.compile(r'(..\s){5}y c')
In English, there will be a 2 characters followed by a space that occurs exactly 5 times. Next to this, the characters “y c” will appear.
Hence, this was the regex pattern I used for the scraping process.
I won’t be going through all the patterns that are needed but you get the idea.
Web scraping is kind of hackish and filled with hard-coded tags.
Anyway, I rinsed and repeated the above steps for everything that I needed to extract.
Let’s move on to the next step.
Step 2 — Define the logical flow in which the script will follow
Recall that the goal is to get all TDS post titles.
We know from step 1 what we want to extract and how to do it.
What we need now is to know how to dynamically extract post titles from all writer profile pages.
This means loading up each writers’ profile and initiate the scraping process.
Keyword “each”.
How do we go about it?
This a three-step process:
We first run a scraping job to extract out all TDS writer profile links.Then use each link to enter into their profile page to extract out all the post titles they had written before.As a sub-task of (2), for each post “div”, create a new entry for the base DataFrame and append the row to it.
We first run a scraping job to extract out all TDS writer profile links.
Then use each link to enter into their profile page to extract out all the post titles they had written before.
As a sub-task of (2), for each post “div”, create a new entry for the base DataFrame and append the row to it.
Here’s how the code looks like if you apply all 3 steps.
Let’s breakdown what you see in the code snippet above.
First, I called a definition “get_profile_urls(...)” that accepts a link to all TDS writers’ profile.
Once the definition scrapes the URL links for each writer, it returns a python list of them.
url_list =['https://medium.com/@kozyrkov','https://medium.com/@ssrosa','https://medium.com/@neha_mangal','https://medium.com/@parulnith', ...]
Each of these links are then looped through and used as inputs to another definition called “extract_information(...)”.
At gist, this definition initializes a Selenium browser for each URL and uses Selenium to scroll all the way down to the end of a profile before setting off two sub-tasks: “get_post(...)” and “get_writer_profile(...)”.
Here’s how the definition of extract_information(...) looks like:
Note lines 21 to 37:
# Path to webdriver    browser = webdriver.Chrome(webdriver_path)# URL to scrape    browser.get(pub_url)    time.sleep(1)# Get body    elem = browser.find_element_by_tag_name("body")# No. of times to scroll    no_of_pagedowns = 100while no_of_pagedowns:        elem.send_keys(Keys.PAGE_DOWN)        time.sleep(0.5)        no_of_pagedowns-=1
This is Selenium starting up a browser for each profile, defining the “body” of the page and scrolling down the page every 0.5 secs for 100 scrolls.
Yes, the 100 is hard-coded. I figured you would reach the end of each profile within 100 scrolls.
If not, just increase it to 150 scroll and that should do the trick.
Moving on, note how I’m keeping track of error counts here.
When it comes to web scraping, there will be a lot of unexpected things that could occur.
In this case, I faced issues with:
Posts having no claps. Thus I had to account for this.
Posts having no H1 title tags.
Posts H4 tags not being equal across all posts.
and more...
With all these little errors, I used “try and except” to log the error counts and allow the program to continuous run even if there were errors.
If the post did not conform to the data frame columns, that would be an error and I logged the count.
In total I scraped, 5524 post titles with 435 errors.
Yikes.
But it’s okay, I make do with 5524. 😏
Let’s now go through how “get_posts(...)” and “”get_writer_profiles(...)” look like.
This definition extracts all relevant information from each post and creates a new entry for the base data frame and appends the new entry to it.
In addition, any posts that fails the extraction process is ignored and an error counter is kept.
Here’s how the code looks like:
Note line 25 and 26:
ends_with_yc = re.compile(r'(..\s){5}y c')    for row in page_content.find_all('div', class_=ends_with_yc):
Recall that I am using a regex match to find the location of each post.
The reason why I’m using Beautiful Soup and not Selenium to do the extraction is because of the matching of the class tag.
As mentioned earlier, there seems to be a bug in Selenium where the matching of class tags aren’t working as they should.
In this project, it was crucial to use the class tag to locate where the relevant “div” tags were.
The next thing I would like to call to attention is in line 63:
dp_rt_tag = row.find_all('span')[3].find('div')
Sometimes it is really helpful to find a parent tag to search it’s child tags.
In line 63, the parent tag was “span” and the child tag, “div”.
You see, web pages tend to be hierarchical in nature and it’s sometimes easier to extract things out knowing the parent tag.
Here’s an example of the hierarchy I’m referring to.
In the image above, the parent tag could be “section” that has a lot of children tags — “div”, “figure”, “h1”, “h2”etc.
Besides extracting out post titles, I also took the liberty of extracting the name of writers and whether they were top writers or not.
This is essentially what this definition does. Pretty straight forward.
I used Selenium for the extraction.
Note the different syntax.
Note that with Selenium, to find a tag, you would run:
match_tag = browser.find_element_by_tag_name("p")ORmatch_tag = browser.find_elements_by_tag_name("p")
Subtle difference between the words “element” vs “elements”.
In Beautiful Soup, the equivalent will be:
match_tag = browser.find("p")ORmatch_tag = browser.find_all("p")
To find a class in Selenium, you would write:
match_tag = browser.find_element_by_class_name("go.gp.gp.gr.gs.y.c")
In Beautiful Soup:
match_tag = browser.find('div', class_="go gp gp gr gs y c")
But finding “class ”tags in Selenium doesn’t always work.
What’s weird is that it works for some class tags but not others. 😅
It worked for the Google example in the introduction didn’t it?
Very strange indeed...
After running the codes you have seen, the final extracted results will have looked like this:
I filtered for TDS as the publisher.
For the writers’ profile:
If you have been following so far, notice how I’ve skipped over one definition? 😉
I did not mention anything about the “get_profile_url(...)” definition.
I skipped this definition so you can get to try it out yourself.
It’s very straight forward compared to the other definitions I shared.
Give it a shot yourself and see how Selenium or Beautiful Soup works!
If all else fails, my scripts and notebooks are in my GitHub page made available below.
You can check them out once you have tried the extraction process yourself.
GitHub link for this post:
github.com
Well, that’s it!
In this post, I’ve given you an introduction to how Selenium and Beautiful Soup works with simple exercises.
Then I gave you the codes to try web scraping on a more realistic set of data. i.e. TDS web page
Note that this scraper can work on any publication not just TDS and in some sense, can be the basis for other web scraping projects you got.
Hopefully this post has given you some insight to web scraping.
With that, I’ll see you in the next post!
Bye!
LinkedIn Profile: Timothy Tan