A Step by Step Guide to Web Scraping in Python | by Sara A. Metwalli | Towards Data Science

As data scientists, we are always on the look for new data and information to analyze and manipulate. One of the main approaches to find data right now is scraping the web for a particular inquiry.
When we browse the internet, we come across a massive number of websites, these websites display various data on the browser. If we, for some reason want to use this data for a project or an ML algorithm, we can — but shouldn’t — gather this data manually. So, we will copy the sections we want and paste them in a doc or CSV file.
towardsdatascience.com
Needless to say, that will be quite a tedious task. That’s why most data scientists and developers go with web scraping using code. It’s easy to write code to extract data from a 100 webpage than do them by hand.
Web Scraping is the technique used by programmers to automate the process of finding and extracting data from the internet within a relatively short time.
The most important question when it comes to web scraping, is it legal?
Short answer, yes.
The more detailed answer, scraping publically available data for non-commercial purposes was announced to be completely legal in late January 2020.
You might wonder, what does publically available mean?
Publically available information is the information that anyone can see/ find on the internet without the need for special access. So, information on Wikipedia, social media or Google’s search results are examples of publically available data.
Now, social media is somewhat complicated, because there are parts of it that are not publically available, such as when a user sets their information to be private. In this case, this information is illegal to be scraped.
One last thing, there’s a difference between publically available and copyrighted. For example, you can scrap YouTube for video titles, but you can’t use the videos for commercial use because they are copyrighted.
There are different programming languages that you can use to scrape the web, and within every programming language, there are different libraries to achieve the same goal.
So, what to use?
In this article, I will use Python, Requests, and BeautifulSoup to scrap some pages from Wikipedia.
To scrap and extract any information from the internet, you’ll probably need to go through three stages: Fetching HTML, Obtaining HTML Tree, then Extracting information from the tree.
We will use the Requests library to fetch the HTML code from a specific URL. Then, we will use BeautifulSoup to Parse and Extract the HTML tree, and finally, we will use pure Python to organize the data.
Before we get scraping, let’s revise HTML basics quickly. Everything in HTML is defined within tags. The most important tag is <HTML> which means that the text to follow is HTML code.
In HTML, each opened tag must be closed. So, at the end of the HTML file, we need a closure tag </HTML>.
Different tags in HTML means different things. Using a combination of tags, a webpage is represented. Any text enclosed between an open and close tag is called inner HTML text.
If we have multiple elements with the same tag, we might — actually, always — want to differentiate between them somehow. There are two ways to do that, either through using classes or ids. Ids are unique, which means we can’t have two elements with the same id. Classes, on the other hand, are not. More than one element can have the same class.
Here are 10 HTML tags you will see a lot when scraping the web.
Awesome, now that we know the basics, let’s start up small and then build up!
Our first step is to install BeautifulSoup by typing the following in the command line.
pip install bs4
To get familiar with scraping basics, we will consider an example HTML code and learn how to use BeautifulSoup to explore it.
BeautifulSoup doesn’t fetch HTML from the web, it is, however, extremely good at extracting information from an HTML string.
In order to use the above HTML in Python, we will set it up as a string and then use different BeautifulSoup to explore it.
Note: if you’re using Jupyter Notebook to follow this article, you can type the following command to view HTML within the Notebook.
from IPython.core.display import display, HTMLdisplay(HTML(some_html_str))
For example, the above HTML will look something like this:
Next, we need to feed this HTML to BeautifulSoup in order to generate the HTML tree. HTML tree is a representation of the different levels of the HTML code, it shows the hierarchy of the code.
The HTML tree of the above code is:
To generate the tree, we write
some_html_str = """<HTML>    <HEAD>        <TITLE>My cool title</TITLE>    </HEAD><BODY>    <H1>This is a Header</H1>    <ul id="list" class="coolList">        <li>item 1</li>        <li>item 2</li>        <li>item 3</li>    </ul></BODY></HTML>"""#Feed the HTML to BeautifulSoupsoup = bs(some_html_str)
The variable soup now has the information extracted from the HTML string. We can use this variable to obtain information from the HTML tree.
BeautifulSoup has many functions that can be used to extract specific aspects of the HTML string. However, two functions are used to most: find and find_all.
The function find returns only the first occurrence of the search query, while find_all returns a list of all matches.
Say, we are searching for all <h1> headers in the code.
As you can see, the find function gave me the <h1> tag. With the tags and all. Often, we only want to extract the inner HTML text. To do that we use .text .
That was simply because we only have one <h1> tag. But what if we want to look for list items — we have an unordered list with three items in our example — we can’t use find. If we do, we will only get the first item.
To find all the list items, we need to use find_all.
Okay, now that we have a list of items, let’s answer two questions:
1- How to get the inner HTML of the list items?
To obtain the inner text only, we can’t use .text straight away, because now we have a list of elements and not just one. Hence, we need to iterate over the list and obtain the inner HTML of each list item.
2- What if we have multiple lists in the code?
If we have more than one list in the code — which is usually the case — we can be precise when searching for elements. In our example, the list has id=’list’ and class=’coolList’. We can use this — both or just one — with the find_all or find functions to be precise and get the information we want.
One thing to note here is the return of the find or find_all functions are BeautifulSoup objects and those can be traversed further. So, we can treat them just like the object obtained directly from the HTML string.
Complete code for this section:
We can traverse the HTML tree using other BeautifulSoup functions, like children, parent, next, etc.
Let’s consider a more realistic example, where we fetch the HTML from a URL and then use BeautifulSoup to extract patterns and data.
We will start by fetching one webpage. I love coffee, so let’s try fetching the Wikipedia page listing countries by coffee production and then plot the countries using Pygal.
towardsdatascience.com
To fetch the HTML we will use the Requests library and then pass the fetched HTML to BeautifulSoup.
If we opened this wiki page, we will find a big table with the countries, and different measures of coffee production. We just want to extract the country name and the coffee production in tons.
To extract this information, we need to study the HTML of the page to know what to query. We can just highlight a country name, right-click, and choose inspect.
Through inspecting the page, we can see that the country names and the quantity are enclosed within a ‘table’ tag. Since it is the first table on the page, we can just use the find function to extract it.
However, extracting the table directly will give us all the table’s content, with the table header — the first row of the table — and the quantity in different measures.
So, we need to fine-tune our search. Let’s try it out with the top 10 countries.
Notice that to clean up the results, I used string manipulation to extract the information I want.
towardsdatascience.com
I can use this list to finally plot the top 10 countries using Pygal.
Complete code for this section:
Wow, that was a lot! 😃
But, we yet to write code that scraps different webpages.
For this section, we will scrap the wiki page with the best 100 books of all time, and then we will categorize these books based on their genre. Trying to see if we can find a relation between the genre and the list — which genre performed best.
The wiki page contains links to each of the 100 books as well as their authors. We want our code to navigate the list, go to the book wiki page, extract info like genre, name, author, and publishing year and then store this info in a Python dictionary — you can store the data in a Pandas frame as well.
So, to do this we need a couple of steps:
Fetch the main URL HTML code.Feed that HTML to BeautifulSoup.Extract each book from the list and get the wiki link of each book.Obtain data for each book.Get all books data, clean, and plot final results.
Fetch the main URL HTML code.
Feed that HTML to BeautifulSoup.
Extract each book from the list and get the wiki link of each book.
Obtain data for each book.
Get all books data, clean, and plot final results.
Let’s get started...
Step #1: Fetch main URL HTML code
url = 'https://en.wikipedia.org/wiki/Time%27s_List_of_the_100_Best_Novels'page = rq.get(url).text
Step #2: Feed that HTML to BeautifulSoup
soup = bs(page)
Step #3: Extract each book from the list and get the wiki link of each book
rows = soup.find('table').find_all('tr')[1:]books_links = [row.find('a')['href'] for row in rows]base_url = 'https://en.wikipedia.org'books_urls = [base_url + link for link in books_links]
Step #4: Obtain data for each book
This is the most lengthy and important step. We will first consider only one book, assume it’s the first one in the list. If we open the wiki page of the book we will see the different information of the book enclosed in a table on the right side of the screen.
Going through the HTML we can see where everything is stored.
To make things easier and more efficient, I wrote custom functions to extract different information from the book’s wiki.
Now, that we have these cool functions, let’s write a function to use these functions, this will help us with the automation.
In this function, I used the try..except formate to avoid crashing if some of the book's info is missing.
Step #5: Get all books data, clean, and plot final results
We have all we need to automate the code and run it.
One last thing to note: It is legal to scrap Wikipedia, however, they don’t like it when you scrap more than one page each second. So we will need to add pauses between each fetch to avoid breaking the server.
Data collected! this will take 100 seconds to finish, so feel free to do something else while you wait 😉
Finally, let’s clean the data, get the genre count, and plot the results.
And we are done!
I have to say, collecting data is not always a 100% accurate, as you can see in the plot, the longest bar belongs to the ‘None’ value. Which means one of two things
Either the wiki page didn’t include the book’s genre.Or, the code for that specific book is different than the rest.
Either the wiki page didn’t include the book’s genre.
Or, the code for that specific book is different than the rest.
That’s why after automating the data collection, we often go through the weird and unusual results and recheck them manually.
Web scraping is one of the essential skills a data scientist needs. And it can’t be any easier than with using Python, Requests, and BeautifulSoup.
We can never trust full automation, sometimes we will need to go through the final result a recheck for abnormal information manually.
The full code for the books section: