Scraping from all over Wikipedia. How to automatically scrape from... | by Kate Christensen | Towards Data Science

Last week I wrote about how to scrape data from a table on Wikipedia (here’s the link to get caught up). In the article, I scraped data from a table on this page, which had the contestants’ name, age, occupation, and where they were from season one of the Great British Bake Off. The end result was the following dictionary:
{'Annetha Mills': [30, '/wiki/Essex'], 'David Chambers': [31, '/wiki/Milton_Keynes'], 'Edward "Edd" Kimber': [24, '/wiki/Bradford'], 'Jasminder Randhawa': [45, '/wiki/Birmingham'], 'Jonathan Shepherd': [25, '/wiki/St_Albans'], 'Lea Harris': [51, '/wiki/Midlothian'], 'Louise Brimelow': [44, '/wiki/Manchester'], 'Mark Whithers': [48, '/wiki/Wales'], 'Miranda Gore Browne': [37, '/wiki/Midhurst'], 'Ruth Clemens': [31, '/wiki/Poynton,_Cheshire']}
As you can see, the contestants’ full names are the keys and the values are a list containing their age and a url snippet containing the url of the wikipedia page about their hometown. Now, I could have easily just gotten the name of the town and called it a day. However, that wouldn’t be helpful if I intend to run a model of some kind on the data I scrape. When you’re gathering data to use for a model, the more actual numerical data you can collect, the better. Therefore, instead of just collecting the name of the place, it would be more useful to collect some statistics about the place, like population or density.
Of course, that information isn’t directly on the GBBO wiki because that’s not relevant to that specific page, and that’s why they link to the page of the town itself. Therefore, this means if we want to collect that information, we have to go to each page and scrape that information ourselves. You could go through every single town page and manually do that, but that’ll likely get tedious fast. That’s why I worked to develop a script that automatically goes to each page and scrapes that information. This way, the actual execution goes a lot faster and is a lot more efficient.
To start off, here’s the full function I wrote to scrape town population and density:
Now we’re going to go back in there and explain what’s actually going on here.
Line two I have the following dictionary:
stats_dict = {'density':None, 'pop':None}
If you see at the end, the second item in the list that is returned is stats_dict. By having that dictionary with empty values, we make sure that in the event something is weird on the page being scraped from, we still get something, and from there we can troubleshoot individual items with issues.
Line four is the following:
if type(url_snippet) == str:
This ensures that the value being passed through is a string, because if it isn’t, the rest of the function obviously won’t work. Lines 5–8 are the following
area_url = 'https://en.wikipedia.org{}'.format(url_snippet)          area_page = requests.get(area_url)         area_soup = BeautifulSoup(area_page.content)
These create the full url, request the url, and make the soup that we will use for scraping information.
The next line is the for loop that all the population and density finding goes under
for item in area_soup.find_all('th'):
This essentially says to look at all of the items tagged ‘th,’ which is where population and density info is typically put under. Lines 11–17 are about finding the density info and lines 19–26 are about finding the population info. Here’s an explanation of how I found the density info:
Lines 11 and 12 are the following:
if 'Density' in item.text:                    if item.find_next_sibling().text:
This is following the initial for loop looking at all the items tagged ‘th’, so this says if the text for something tagged as a header is ‘Density’, meaning look for the header labeled density. The next line is saying if the sibling of the thing labeled density has text, do the following.
Note: through this process, you’re going to notice there’s a lot of if statements. This is because there are occasionally slight differences in how different pages are formatted. To a page viewer, this doesn’t really diminish their experience a whole lot. However, when you’re working with a script, a slightly different arrangement could mean an error for you. Therefore, you have to do a lot of, “if this thing exists, do this” instead of just telling the function to do something.
The following lines are:
density_str = item.find_next_sibling().text                    for elem in density_str.split():                            if "/km" in elem:
Going back to that last if statement, this first line here is saying, “now that we know this thing exists, that item is the string with the density info.” The next two lines split up the text into all its individual parts, and then takes the section that has the person per kilometer info, since we want to make sure our units are consistent for the numbers we get. The following two lines actually extract the data (the first line spills over a bit):
d = int([item for item in re.split("(\d*,?\d*)",elem) if len(item)>1][0].replace(',',''))                            stats_dict['density'] = d
The first line is what actually accesses the number. In short, it uses regex to split the text, if that item can be split, then takes the first item in that list, which is the number itself, then it replaces the comma in the text with nothing, then turns that final number from a string into an integer. The next line goes back to that initially empty dictionary and updates the density entry with that number collected in the previous line.
And voila, we have our density! Now on to the population, which I found to be a bit more tricky because the formatting varied a touch more than it did with density.
if 'Population' in item.text:
Like density, we start out looking for item tagged as a header whose text is ‘Population.’ However, after this, we start to diverge from the path we took to obtain town density. Before we get into the code itself, I want to demonstrate exactly why it may look a little more complicated than you might initially think it should:
Here is the Wikipedia page for Essex, hometown of Annetha Mills:
Here we see essentially what we saw with density, the number we’re searching for is in the sibling’s text. However, that’s not how it’s consistently formatted. If we look at Bradford, hometown of Edward “Edd” Kimber:
The number of the population is in the parent’s sibling text. Therefore our next lines are a bit different:
if item.find_next_sibling() is None:     stats_dict['pop'] = re.split('(\d*,\d*)',item.parent.find_next_sibling().text)[1]
Essentially, if the item doesn’t have a sibling, go up to the parent, find the parent’s sibling, then get the text from the parent’s sibling, split it up, and find the number. That number is now the value for the population, or ‘pop’, entry in the stats_dict. Then the next condition is the following:
elif item.find_next_sibling().text.startswith('%'):     pass
In this case, if the item has a sibling, but that sibling’s text starts with a percent sign, just pass over it because that won’t have the info you’re looking for. Finally, if the item has a sibling, and that sibling’s text does not start with a percent sign we have the following:
else:     pop_str = item.find_next_sibling().text     stats_dict['pop']= int(pop_str.split()[0].replace(',',''))
This is not nearly as complicated as before. Basically, take the item’s sibling, get its text (pop_str), split that text, take the first item in the split list (the item that contains the number), take out the comma in the number, then turn that into an integer. That is now assigned as the ‘pop’ value in the stats_dict. At long last we have our return statement:
return [url_snippet.split('/')[-1], stats_dict]
This returns a list where the first item is just the name of the place from the initial url snippet. I keep this name in in the event density or pop come out as a None value. If that does happen, I can go look at the page myself and scrape it manually. The second item in that list is the stats_dict that contains the density and population info.
Now we that we have a function, it’s time to use it.
Our final product from my last article, the dictionary shown at the beginning of this one, will be named “contestant_name_age_town.”
The first three lines say for each list in the values of the dictionary, the town is the last item in that list, and that list should now be extended using the area_stats function, which will return a list of the place name and stats about that place. Finally, now that we have all the area stats, we can take out the url snippet. All of this turns this dictionary:
{'Annetha Mills': [30, '/wiki/Essex'], 'David Chambers': [31, '/wiki/Milton_Keynes'], 'Edward "Edd" Kimber': [24, '/wiki/Bradford'], 'Jasminder Randhawa': [45, '/wiki/Birmingham'], 'Jonathan Shepherd': [25, '/wiki/St_Albans'], 'Lea Harris': [51, '/wiki/Midlothian'], 'Louise Brimelow': [44, '/wiki/Manchester'], 'Mark Whithers': [48, '/wiki/Wales'], 'Miranda Gore Browne': [37, '/wiki/Midhurst'], 'Ruth Clemens': [31, '/wiki/Poynton,_Cheshire']}
into this dictionary:
{'Annetha Mills': [30, 'Essex', {'density': 427, 'pop': 1477764}], 'David Chambers': [31, 'Milton_Keynes', {'density': 2584, 'pop': 229941}], 'Edward "Edd" Kimber': [24, 'Bradford', {'density': 1290, 'pop': '539,776'}], 'Jasminder Randhawa': [45, 'Birmingham', {'density': 4262, 'pop': '1,141'}], 'Jonathan Shepherd': [25, 'St_Albans', {'density': 8140, 'pop': 147373}], 'Lea Harris': [51, 'Midlothian', {'density': 260, 'pop': '91,340'}], 'Louise Brimelow': [44, 'Manchester', {'density': 4735, 'pop': '552,858'}], 'Mark Whithers': [48, 'Wales', {'density': 148, 'pop': '278,000'}], 'Miranda Gore Browne': [37, 'Midhurst', {'density': 1467, 'pop': 4914}], 'Ruth Clemens': [31, 'Poynton,_Cheshire', {'density': 1089, 'pop': 14260}]}
Now we actually have some useful area stats and we didn’t have to individually comb through every single page to find that info!