NLP Part 1| Scraping the Web using BeautifulSoup and Python | by Kamil Mysiak | Towards Data Science

Data is at the core of any data science project, yet often we take for granted the availability of data especially when it arrives neatly in a SQL database or better yet in our inbox.
That said, sometimes the data you’re looking for isn’t readily available due to its specific nature. One possible solution to this problem is the idea of Web Scraping or the extraction of information from a specific website by carefully reading its HTML. For example, let’s say you’re planning a vacation and you’re on the lookout for when the airfare goes on sale. Yes, you can browse the same travel site each hour hoping the price would drop but a more efficient method would be to scrape the travel site each hour and have an output file provide you with the most current ticket prices.
Many websites do not look kindly to having their data scraped, especially if it contains identifiable user information (i.e. Facebook, Linkedin, etc.). Please be considerate of what data you choose to scrape and how often.
This short tutorial is the first part of a 3-part series on Natural Language Processing (NLP). In this series, we’ll explore techniques of scraping website data, pre-processing and getting our data ready for analysis, and finally gleaning insights from our NLP data.
NLP Part 2
NLP Part 3
In this example, let’s take a stab at scraping indeed.com but specifically company reviews. Let’s target the employees’ ratings, review titles, review descriptions along with the pros and cons.
Before we can actually begin scraping information we need to familiarize ourselves with the basic structure of HTML as we’ll actually be using HTML tags to identify what information we wish to scrape.
We can access a website’s HTML by opening the developer tools in your current browser. For example, Firefox (options → Web Developer → Inspector). All those blue “div” tags, arrows, classes and ids is the HTML for the website you’re currently on.
Before we examine the HTML for indeed.com let’s review the basic structure using the example below.
HTML describes the underlying structure of a website. In other words, it identifies that the website will have a header, multiple paragraphs, an embedded video, a closing footer, etc. HTML will not describe how these components will be arranged, their style, size, color, etc.
HTML code is hierarchical in nature and indented tags (ie. <div>, <ul>, etc.) identify each new level in the hierarchy. For example, the “<html>” tag is at the top of the hierarchy (also called a container) and it contains all other elements. The body tag is especially important as it contains the vast majority of the information visible on a website. As you can see the body tag houses a variety of other nested tags which in our example stores a paragraph which then contains a link, a review title and the actual review text “Pretty good”. It is important to be able to read and understand the nested trail of tags which lead us to the actual review text “Pretty good” because this is the information we want to scrape. Let’s turn our attention to the python code which drills through this hierarchical structure of tags to pull out the text.
First, we’ll need to import the required libraries.
from bs4 import BeautifulSoupimport lxmlimport requestsimport pandas as pdimport numpy as np
The imported “request” library has a get() function which will request the indeed.com server for the content of the URL and store the server’s response in the “base_url” variable. If we print the “base_url” variable we’ll actually see the entire HTML for the page.
base_url = requests.get('https://www.indeed.com/cmp/Google/reviews?fcountry=ALL&start=', timeout=5)print(base_url.text)
Let’s begin by defining a function named “parse” requiring one argument which will be the actual URL of the page we are attempting to parse/scrape. Next, we’ll use the BeautifulSoup class creator to parse the content (HTML code) of the provided website. We’ll use the “lxml” parser just in case the HTML is not perfectly formed. For more information about the different parsers available with BeautifulSoup visit this link.
Please keep in mind websites will often adjust their HTML script by changing names or parent/child relationships of items in containers. These changes will require you to adjust your function. This function was adjusted 6/20 to fix the newly implemented HTML script for indeed.com
def parse(full_url):    page_content = BeautifulSoup(full_url.content, 'lxml')    containers = page_content.findAll('div',                  {'class':'cmp-Review-container'})    df = pd.DataFrame(columns =          ['rating', 'rating_title',  'rating_description',                         'rating_pros', 'rating_cons'])        for item in containers:        try:            rating = item.find('div',                      {'class': 'cmp-ReviewRating-text'})                     .text.replace('\n', '')        except:            rating = None        try:            rating_title = item.find('div',                            {'class': 'cmp-Review-title'})                           .text.replace('\n', '')        except:            rating_title = None        try:            rating_description = item.find('span',                                  {'itemprop': 'reviewBody'})                                 .text.replace('\r', '. ')        except:            rating_description = None        try:            rating_pros = item.find('div',                           {'class': 'cmp-ReviewProsCons-prosText'})                          .text.replace('\n', '')        except:            rating_pros = None        try:            rating_cons = item.find('div',                           {'class': 'cmp-ReviewProsCons-consText'})                          .text.replace('\n', '')        except:            rating_cons = None        df = df.append({'rating': rating,              'rating_title': rating_title,              'rating_description': rating_description,             'rating_pros': rating_pros,              'rating_cons': rating_cons}, ignore_index=True)    return df
Next, we’ll need to examine the HTML more closely in order to identify which root container (ie. which parent tag) houses the child/nested tags which contain the information we which to scrape. Let’s navigate to “https://www.indeed.com/cmp/Google/reviews?fcountry=ALL&start=” as we’ll be scraping employees’ reviews of google. Open your developer tool in order to observe the HTML for the website. Notice that once you have access to the website’s HTML, moving your mouse cursor around the website causes specific areas to become highlighted and the HTML seems to change right along as you move the cursor. As you move your cursor your developer tools automatically takes you to the HTML section of the highlighted section on the webpage. This is extremely helpful as we can quickly identify which parts of the HTML code we need to examine in more detail.
If you recall, in this tutorial we are interested in scraping the employees’ overall rating, review title, review description, pros and cons. We need to identify which HTML tag is the container or home for all this information. By moving our mouse cursor to the appropriate location we see that all the information we wish to scrape is contained neatly inside one root element. By examining the HTML below we can see that the “<div” tag contains all the information we need in a class attribute named “cmp-Review-container”. Now move to a different review and you’ll see the same class attribute named “cmp-Review-container” which stores that review’s data.
Therefore, we’ll use the “findall()” method to extract all the “div” containers that have a class attribute of “cmp-Review-container”. Next, we are going to create an empty pandas dataframe named “df” to which we’ll append the scraped data.
def parse(full_url):    page_content = BeautifulSoup(full_url.content, 'lxml')    containers = page_content.findAll('div',                  {'class':'cmp-Review-container'})    df = pd.DataFrame(columns =          ['rating', 'rating_title',  'rating_description',                         'rating_pros', 'rating_cons'])        for item in containers:                try:            rating = item.find('div',                      {'class': 'cmp-ReviewRating-text'})                     .text.replace('\n', '')        except:            rating = None        try:            rating_title = item.find('div',                            {'class': 'cmp-Review-title'})                           .text.replace('\n', '')        except:            rating_title = None        try:            rating_description = item.find('span',                                  {'itemprop': 'reviewBody'})                                 .text.replace('\r', '. ')        except:            rating_description = None        try:            rating_pros = item.find('div',                           {'class': 'cmp-ReviewProsCons-prosText'})                          .text.replace('\n', '')        except:            rating_pros = None        try:            rating_cons = item.find('div',                           {'class': 'cmp-ReviewProsCons-consText'})                          .text.replace('\n', '')        except:            rating_cons = None        df = df.append({'rating': rating,              'rating_title': rating_title,              'rating_description': rating_description,             'rating_pros': rating_pros,              'rating_cons': rating_cons}, ignore_index=True)    return df
Now that we have identified the container for all the data we wish to scrape let’s dive a bit deeper into the HTML to identify the elements which house the actual data. First, the review rating is once again housed in the “cmp-Review-container” container but drilling down a few layers we find the “<div” tag with a class attribute of “cmp-ReviewRating-text which actually stores the “5.0” rating. Let’s make note of the tag and class attribute which stores this data as we’ll need this information for our python script below. We repeat the process for all remaining pieces of data which wish to extract.
Here’s a challenge for you. Why are we replacing the carriage return ( “\r”) with (“. “) for our rating_descriptions? Hint: What is the difference between reviews titled “Great work environment” and “Nice place to work”? Post your answers in the comments below! 😉
Once we have identified the appropriate tags for the data let’s turn our attention back to the python code. We can use a try/except block inside a for-loop to search the container for the identified tags using a find() method. We use find() instead of findall() because we only want to return the first match due to the fact we are using a for-loop. Finally, we append the scraped data back into the empty dataframe we created earlier.
def parse(full_url):    page_content = BeautifulSoup(full_url.content, 'lxml')    containers = page_content.findAll('div',                  {'class':'cmp-Review-container'})    df = pd.DataFrame(columns =          ['rating', 'rating_title',  'rating_description',                         'rating_pros', 'rating_cons'])        for item in containers:                try:            rating = item.find('div',                      {'class': 'cmp-ReviewRating-text'})                     .text.replace('\n', '')        except:            rating = None        try:            rating_title = item.find('div',                            {'class': 'cmp-Review-title'})                           .text.replace('\n', '')        except:            rating_title = None        try:            rating_description = item.find('span',                                  {'itemprop': 'reviewBody'})                                 .text.replace('\r', '. ')        except:            rating_description = None        try:            rating_pros = item.find('div',                           {'class': 'cmp-ReviewProsCons-prosText'})                          .text.replace('\n', '')        except:            rating_pros = None        try:            rating_cons = item.find('div',                           {'class': 'cmp-ReviewProsCons-consText'})                          .text.replace('\n', '')        except:            rating_cons = None        df = df.append({'rating': rating,              'rating_title': rating_title,              'rating_description': rating_description,             'rating_pros': rating_pros,              'rating_cons': rating_cons}, ignore_index=True)    return df
We are not done just yet because if you were to execute the “parse()” function you will obtain a dataframe with only 20 records and that’s due to the fact we have only scraped one page.
In order to scrape all remaining pages of reviews, we first create a new empty dataframe to collect all the reviews as we iterate over all the pages. Next, we initiate a counter variable of 20 because there are 20 reviews per page. Next, we create a while-loop which will iterate until the number of reviews is equal to or larger than 4000. Why 4000? There are almost 4000 individual reviews across all the pages (*at the time this article was written*). Next, we add the increment of 20 to the end of the base_url as the while-loop iterates through each page. Adding an increment of 20 to the end of the base_url will create a new url specific for the page you wish to visit. For example, “https://www.indeed.com/cmp/Google/reviews?fcountry=ALL&start=40” will take us to the second page of the reviews. Next, we once again request and get the entire HTML for the page the while-loop is iterating over. We apply our parse() function on the page and append the newly scraped reviews into our dataframe. Finally, we increase the counter by 20 in order for the while-loop to iterate on the next page.
base_url = 'https://www.indeed.com/cmp/Google/reviews?fcountry=ALL&start='all_reviews_df = pd.DataFrame(columns = ['rating', 'rating_title', 'rating_description','rating_pros', 'rating_cons'])num_reviews = 20# you can adjust this number on how many reviews you which to scrapewhile num_reviews < 3000:          full_url = base_url + str(num_reviews)        get_url = requests.get(full_url, timeout=5)              partial_reviews_df = parse(get_url)       all_reviews_df = all_reviews_df.append(                     partial_reviews_df, ignore_index=True)         num_reviews += 20
All that’s left to do is move our dataframe to a csv file.
all_reviews_df.to_csv('indeed_scrape.csv')
I hope you enjoyed this tutorial. I would love your feedback, feel free to comment below.
Don’t forget the challenge!