Data Science Essentials: Scraping Data From the Web
By: Eijaz Allibhai
Tags: Python, Sql, Machine Learning, Data Science, Data Analysis

Web scraping is an important skill for any data scientist to have in their toolbox

image credit: link

Web scraping is an important skill for any data scientist to have in their toolbox. Web scraping can be used to collect data about products for sale, user posts, images, and pretty much anything else that is useful on the web. If using a web scraper, just keep in mind that some websites do not want anyone to scrape their sites and when a website updates, you may need to alter your web scraper.

In this tutorial, we will be building a web scraper to collect data on the undergraduate courses taught at City, University of London, where I completed my Master’s Degree in Data Science.

Note: For any American students reading this, a course in the UK is equivalent to your major in the US (it’s not just one class).

Specifically, we will scrape the course name, degree name, the school the course is part of, course duration, the course code.

To begin, let’s check the url of the site we’re going to scrape. There are 7 pages of courses, so we will need to loop through all 7 pages. To do this, we can use a for loop.

In the url, the page numbers update as 01, 11, 21, 31, etc. so we will update the url as we work through the loop. We will use the urllib library to read in the url and use the Beautiful Soup library to help access the information we are looking to scrape.

from bs4 import BeautifulSoup

import urllib.request as ur



#loop through the 7 pages of undergraduate courses

for i in range (0,6):



#read and open url to scrape

urlToScrape = "https://www.city.ac.uk/courses?level=Undergraduate&p=" + str(i * 10 + 1)

r = ur.urlopen(urlToScrape).read()

soup = BeautifulSoup(r, "lxml")

To get to the information, we need to inspect the webpage to check out the HTML elements of the page in which the information is wrapped in.

To inspect the webpage, open the webpage on Google Chrome, right-click on your mouse, and select ‘Inspect’. You will see a window with all the HTML elements of the webpage. If you are not familiar with this, take a few minutes to play around with it. You can highlight a part of the webpage, then right-click and choose ‘Inspect’ to see where the highlighted section is included in the HTML.

Each block of information about a course is wrapped in a ‘div’ and has a ‘class’ of course-finder__results__item course-finder__results__item — undergraduate. We are going to use the Beautiful Soup library to find all instances of this class and store it to ‘courseList’.

#attributes for each course

courseList = soup.find_all('div', attrs={'class': 'course-finder__results__item course-finder__results__item--undergraduate'})

To get the information from each course, we can loop through ‘courseList’. Let’s highlight the first course name on the webpage and inspect it. It’s wrapped in a ‘div’ and has a ‘class’ of col-sm-24 col-md-18 col-lg-20. We have to find that class within ‘courseList’ and save it to ‘courseNameElement’.

To scrape the course name text, we can just search for the text within ‘courseNameElement. Since the course name is a link, the text is wrapped in an ‘a’ tag, so you can specify that in the ‘find’ method.

Sometimes, some of the information we are looking for may not be there for every course. If this is the case, our program will stop running and throw an error. To bypass this, we can wrap our code in a try except block. In this case, the program will try to find the relevant information. If it is not there, instead of throwing an error, it will undertake an action that we specify. In this instance, we will save an empty string to the variable.

#loop through each course

for courseListItem in courseList:



#get course name

try:

courseNameElement = courseListItem.find('div', attrs={'class': "col-sm-24 col-md-18 col-lg-20"})

courseName = courseNameElement.find('a').text

except Exception as e:

courseName = ''

Before you check the code below, see if you can find the HTML elements that the degree name, the school the course is part of, course duration, the course code are located in by using the inspect method that we just used. Remember that you can highlight certain parts of the webpage and expand/close the arrows in the inspect window.

Here is the code to scrape the rest of the course information:

#get degree name

try:

degreeTypeElement = courseListItem.find('div', attrs={'class': "course-finder__results__item__award"})

degreeName = degreeTypeElement.text

except Exception as e:

degreeName = ''



#get school the course is part of

try:

courseSchoolElement = courseListItem.find('div', attrs={'class': "course-finder__results__item__md course-finder__results__item__md--school"})

courseSchool = courseSchoolElement.find('a').text

except Exception as e:

courseSchool = ''



#get course duration

try:

courseDurationElement = courseListItem.find('div', attrs={'class': "course-finder__results__item__md course-finder__results__item__md--duration"})

courseDuration = courseDurationElement.find_all('span')[2].text

except Exception as e:

courseDuration = ''



#get course code

try:

courseCodeElement = courseListItem.find('div', attrs={'class': "course-finder__results__item__md course-finder__results__item__md--code"})

courseCode = courseCodeElement.find_all('span')[2].text

except Exception as e:

courseCode = ''

Lastly, let’s print out our newly scraped course information.

print(courseName) print(degreeName) print(courseSchool) print(courseDuration) print(courseCode) print (“ — — — — — — -

”)

You should have a long list of many undergraduate courses printed. Here is what the results should look like:

Congrats! You have built your own web scraper!

Thanks for reading! You can access the code on my Github here.

If you would like to keep up to date with my data science and machine learning content, follow me!