Extracting webpage information with Python for non-programmers | by Elye | Towards Data Science

Knowing how to extract data from the webpage using a script is helpful. The easiest way to do so that I found is using Python, with the help of lxml library.
The nicest part is, you might not need to do it in your machine. You could put your script on the Python online compiler, and you get some data.
For something simple, we just need a few lines of codes in a single file, then it will do pretty cool things.
Several examples below.
medium.com
medium.com
To make learning more relevant, we’ll take the Wordometer Coronavirus webpage as our source.
If you visit the page, you’ll see something similar to below. I separate the learning into 4 sections, which I called as Exercise each as below.
I have provided the code script (just 70 lines) whereby copy-paste and running it (in the online compiler), you’ll see the below
And select each of the numbers to execute it. This should make learning easier.
All webpage is constructed using HTML.
Let’s get to the most basic structure of HTML, as that’s the basic pattern we need to recognize to extract the information we need.
The most basic structure I could provide is as below.
All of them consist of tag that is wrap with < and >. And it usually with a compliment of it i.e. </tag> ` or it just ends with /> in the tag itself.Within the <tag> it might have one or more attribute with a name given to it. That helps to differentiate tag of the same type.Around the <tag> pair, we could have one (or more) tag that is wrapped within, which I name as <sub-tag> above. You could imaging, sub-tag could have sub-sub-tag and so on.... , which makes up the majority part of the HTML.At times, the tag could wrap a text around it, which normally form the core of the information we are looking for.
All of them consist of tag that is wrap with < and >. And it usually with a compliment of it i.e. </tag> ` or it just ends with /> in the tag itself.
Within the <tag> it might have one or more attribute with a name given to it. That helps to differentiate tag of the same type.
Around the <tag> pair, we could have one (or more) tag that is wrapped within, which I name as <sub-tag> above. You could imaging, sub-tag could have sub-sub-tag and so on.... , which makes up the majority part of the HTML.
At times, the tag could wrap a text around it, which normally form the core of the information we are looking for.
That’s it! Give a pat to yourself. You have mastered HTML (and also XML) structure 🎉
Before we could extract the HTML information, we need to get our script to read the HTML first. There are 2 ways of doing so.
We’re using the request library of Python. Don’t worry, that’s as simple as the line below, then it’s done.
import requests
After that, try getting the website content using the code below.
response = requests.get(url)
To be more robust, in case the url is wrong or the website is down, I add the following
try:  response = requests.get(url)except:  print ('Sorry bad url')   sys.exit(2)if response.status_code != 200:  print ('Sorry invalid response ' + str(response.status_code))  sys.exit(2)
If the request is loaded successfully, then you could get Html content, using the below code, and extract it into a tree.
tree = html.fromstring(response.text)
The reason it is called a tree because if we draw the tags relationship, they look like one.
Sometimes this is needed, as some webpages require credentials to log in, and hence to access it from the script would be more complicated as you need to supply the needed credential along the way.
But since you just need to extract some data from the HTML, you could save it as a .html file and get the script to read from it.
After that, in your python script, you just need to
Open the file and read it as a String (set of letters).Read out the HTML content to form a tree
Open the file and read it as a String (set of letters).
Read out the HTML content to form a tree
inputFile = open(inputfile,"r") fileContent = str(inputFile.readlines())tree = html.fromstring(fileContent)
1 Extracting single tag wrap text. Just to recap, for exercise 1, we want to extract the names of the countries as shown below.
If we check the HTML code for it, we’ll see the below pattern.
<a class=”mt_a” href=”country/spain/”>Spain</a>
To extract the Spain, we’ll just need to identify a tag that has attribute class of "mt_a".
To extract, we just use xpath function for the tree.
extracteditems = tree.xpath(‘//a[@class=”mt_a”]/text()’)print("\n".join(extracteditems))
We’ll get a list of the, where we separate them by a newline (i.e. "\n") and print them out.
We’ll get
USASpainRussiaUKItaly... and many more ...
Easy!
2 Extracting tag and sub-tag wrap text. Just to recap, for exercise 2, we want to extract the population information of each country.
If we check the HTML code for it, we’ll see the below pattern.
<td style=”font-weight: bold; text-align:right”>   <a href=”/world-population/spain-population/”>46,752,556</a> </td><td style=”font-weight: bold; text-align:right”>2,467,761</td><td style=”font-weight: bold; text-align:right”>52,783</td>
We cannot capture <a href=”/world-population/spain-population/”> as the href for each country will be different.
We also cannot capture <td style=”font-weight: bold; text-align:right”>, as there are a lot of other numbers as shown above (e.g. 2,467,761) that we don’t want.
Hence the best is to capture <td style=”font-weight: bold; text-align:right”> that follow by <a> tag.
To extract, we just use xpath function as well for the tree.
extracteditems = tree.xpath(    '//td[@style="font-weight: bold; text-align:right"]//a/text()')print("\n".join(extracteditems))
We’ll get a list of the, where we separate them by a newline (i.e. "\n") and print them out.
330,758,78446,752,556145,926,78167,841,32460,472,650... and many more ...
Easy!
3 Extracting attrib value from a tag. Just to recap, for exercise 3, we want to extract the image URL that the website has.
In HTML, the images URL store as the src attribute of the img tag.
<img src="/img/alert-plus.png" style="height:16px; width:16px" />
To extract it, we first need to extract the img element first, then use a special extraction below i.e. list(map(lambda ...)) to extract each src attribute out i.e. x.attrib.get(‘src’).
The x is a single item of each elements within.
elements = tree.xpath(‘//img’)extracteditems = list(map(lambda x: x.attrib.get(‘src’), elements))print(“\n”.join(extracteditems))
From this, we can now get
/img/worldometers-logo.gif/images/alert.png/img/alert-plus.png... and a few others ...
A little complicated on the list(map(lambda ...)) but it’s an approach that extracts individual items of the elements list.
4Iterate through each tag to extract the sub-tags. Just to recap, for exercise 4, we want to extract the high-level numbers for each continent
To be more specific, as we enter one of the continents, the value is as shown below, that’s what we want to extract.
If we look into the HTML, each of the continents data is grouped with tr that has class attribute of "total_row_world row_continnet".
<tr class="total_row_world row_continent" data-continent="Europe" style="display: none">  <td></td>  <td style="text-align:left;"><nobr>Europe</nobr></td>      <td>1,741,129</td>  <td></td>  <td>160,482</td>  <td></td>  <td>739,811</td>  <td>840,836</td>  <td>12,196</td>  <td></td>  <td></td>  <td></td>  <td></td>  <td></td>  <td style="display:none;" data-continent="Europe">Europe</td></tr>
Let’s extract the "total_row_world row_continnet" as sections of information.
To further extract each item within each section, we create a separate function name extractElements as shown below. We send in x which is each section from the sections.
sections = tree.xpath(    '//tr[@class="total_row_world row_continent"]')extracteditems = list(map(lambda x:     '\"' + str(extractElements(x)) + '\"', sections))print("\n".join(extracteditems))
Now in the function extractElements, we use findall to extract out all the td tag, and again use the list(map(lambda ...)) to separately extract it’s text.
def extractElements(element):    return list(map(lambda x: x.text, element.findall('.//td')))
The output as shown below.
That’s it!
To further illustrate how element.findall(‘.//td’) work, I use a for-loop to individually print out each extracted
def extractElements(element):  for item in element.findall('.//td'):    print('  EXTRACTED: ' + str(item.text))return list(map(lambda x: x.text, element.findall('.//td')))
The output as below
EXTRACTED: 1,741,129EXTRACTED: +22,456EXTRACTED: 160,482EXTRACTED: +1,414EXTRACTED: 739,811... and more ...EXTRACTED: EuropeEXTRACTED: NoneEXTRACTED:
Having learned all these four ways, you now have the basic essential to explore some website of your interest and extract relevant info for your further processing. 🛠
Cheers.
Thanks for reading. You can check out my other topics here.
You could follow me on Medium, Twitter, Facebook, or Reddit for little tips and learning on mobile development, medium writing, etc related topics. ~Elye~