Step-by-Step Tutorial: Web Scraping Wikipedia with BeautifulSoup | by Khuyen Tran | Towards Data Science

My roommate and I had a discussion about her observation of the high depression rate in Sweden. We drew the connection between the depression rate and the lack of sunshine. I decided to support my hypothesis by gathering my own data and analyzing it.
I use Beautiful Soup, an easy-to-use Python tool for web scraping. What I love about this tool is that it is easy to use and understand. The documentation for the library can be found here.
Hypothesis: The fewer hours of sunshine in a country, the higher the rate of depression that the country has.
Data to collect:
Names of Countries
Their depression rate
Their sunshine hours
Resources to use:
https://en.wikipedia.org/wiki/Epidemiology_of_depression
https://en.wikipedia.org/wiki/List_of_cities_by_sunshine_duration
import requestsimport urllib.requestimport timefrom bs4 import BeautifulSoupimport numpy as npimport pandas as pdfrom urllib.request import urlopenurl = 'https://en.wikipedia.org/wiki/Epidemiology_of_depression'html = urlopen(url) soup = BeautifulSoup(html, 'html.parser')
Right-click on the page then go to inspect. Click on the icon shown above to choose the element in the website to inspect.
We are interested in extracting the information from the table. This could be done easily with one line of code. By inspection, we find that the table is under <table> tag.
find_all('table') scans the entire document to look for the tag <table>
tables = soup.find_all('table')
find_all() is an easy and effective method that will be used a lot in this tutorial.
Look like the data in theDALY rate column will be a string when collected because of “,”. Create a function to process the string into an integer by using re.sub(), a method to return the substring that matches the regular expression.
import redef process_num(num):    return float(re.sub(r'[^\w\s.]','',num))
Test the function
num1 = float(re.sub(r'[^\w\s.]','','1,156.30'))num1
Output: 1156.30
Let’s break this down.
Line 6–7: We pay attention that each row is under <tr> tag. This gives us the idea that we will get the rows by finding all the tag <tr>
Line 9–10: Extract rows by finding all <td> tags
Line 14: Since len(cells) in the first row is 1, we set the condition to skip the first row and extract from the other rows.
Line 15: Because cells is a list of 3 cells, we extract the rank using the first cell, country name using the second cell, and DALY rate using the third cell.
Put the data into a pandas.DataFrame
df1 = pd.DataFrame(ranks, index= countriescolumns = ['Rank'])df1['DALY rate'] = ratesdf1.head(10)
Use the same method as above to collect the sunshine hours from the second website. There are 2 differences between the data of the 2 websites to keep in mind:
There are multiple data about the sunshine hours from the same country. Thus we might want to take the average of these data to get the hours represented for each country. To take the average, we simply use 2 dictionaries, country_suns and countwith the same keys but different values.
Some countries in the second data are not available in the first one. Thus, we just want to collect the data from the countries in the first data, that are stored in the variable countries created above.
Outcome:
789.14 3Country: Benin, Sunshine Hours: 263.05515.99 2Country: Togo, Sunshine Hours: 258.0710.25 3Country: Ghana, Sunshine Hours: 236.75866.0500000000001 4Country: Cameroon, Sunshine Hours: 216.51344.03999999999996 2Country: Gabon, Sunshine Hours: 172.021334.54 5Country: Nigeria, Sunshine Hours: 266.91711.91 2Country: Sudan, Sunshine Hours: 355.95336.1 1...
Use the join method to join the 2 data we collected previously.
df2 = pd.DataFrame.from_dict(country_suns,orient='index', columns = ['Sunshine Hours/Year'])df = df1.join(df2)df.info()
Outcome:
<class 'pandas.core.frame.DataFrame'>Index: 192 entries, United States to JapanData columns (total 3 columns):Rank                   192 non-null int64DALY rate              192 non-null float64Sunshine Hours/Year    122 non-null float64dtypes: float64(2), int64(1)memory usage: 11.0+ KB
We realize that there are null values in the last column because some countries in the first source are not in the second source. There are many ways to process missing values. But for now, just drop the rows with the missing values to visualize what we got.
df.dropna(inplace=True)
import matplotlib.pyplot as pltimport seaborn as snssns.scatterplot('Rank', 'Sunshine Hours/Year', data=df)
There seems not to be a strong correlation here. Take a look at the correlation coefficient.
Save the data if you want to work with the data using a different tool.
df.to_csv('wiki-2.csv')
The correlation is not as strong as I thought it would be. But we haven’t taken into account different factors such as the accurateness of the metric to determine to depression rate, the size the countries (the bigger the country, the more variation in sunshine hours/year), GDP, population, etc. More data should be gathered to attain a more accurate result. But before that, feel proud of yourself because you have just learned web scraping, one of the most essential processes in data science.
The Github repository for this tutorial could be found here: https://github.com/khuyentran1401/Web-Scrapping-Wikipedia.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these:
[1] Epidemiology of depression, Wikipedia
[2] List of cities by sunshine duration, Wikipedia
[3]Varun Choudhary, How To Scrape A Website Without Getting Blacklisted (2019), Hackernoon