Scraping COVID-19 data from various websites (Part 3 - end)
By: Reza Dwi Utomo
Tags: Web Scraping, Coronavirus, Beautifulsoup, Selenium, Covid 19

There are 5 buttons, and they all have the same tag and text, i.e. <a></a> and “Detail” respectively. Therefore, we can find their element by using their tag and text. Subsequently, it’s straightforward to get the link of each element. After all external links collected, we can implement Pandas read_html to scrape the table and automatically convert it to a dataframe. We can use the code snippet below.

Cool. Using Pandas read_html really reduces our time and lines of code to scrape the data compared to using Selenium. Now, let’s see the results. There are 5 tables as there are 5 buttons (with their table) as well.

Table from Kecamatan Melaya

Table from Kecamatan Jembrana

Table from Kecamatan Negara

Table from Kecamatan Mendoyo

Table from Kecamatan Pekutatan

Pandas read_html reserves the original table structure of the data, so that all tables above show the same structure as the pop-up tables when each Detail button clicked. Sometimes, this made your life easier in case you wanted to retain the structure for future data processing. But in this case, we don’t want to keep the original structure of the tables. Instead, we have our own table structure. Therefore, for example, to process the tables, we can implement what has been explained and used in the second part of the tutorial.

Scraping data from a map using polygon

In the second section, we’re going to scrape data from the map on a website. The Buleleng website is used this time. If you see it, you’ll find that the only data in the tabular format are the ones with cumulative (in a whole Kabupaten). So, if you’d like to get data in a more specific area, i.e. Kecamatan, you only have one choice to collect such data directly from the map on the webpage. See the gif image below.

Moreover, all data of each Kecamatan on the map are just marked by pins, containing a number and a colour which indicates OTG, ODP, etc. Each pin is located in a specific area of Kecamatan. Therefore, we can say that if any pin is in an area of Kecamatan, such a pin belongs to such a Kecamatan. Each Kecamatan, furthermore, has a colour as well, indicating what zone it is. This understanding will be used to collect the data. So, below are our strategies:

get all available Kecamatan elements scrape the x-y coordinates of each Kecamatan get all available pins’ element scrape the x-y coordinates of each pin examine what pin belongs to what Kecamatan area click each Kecamatan (and adjust the screen of the map (using zoom-in, zoom-out, or dragging), if needed, so that the Kecamatan area is visible and clickable) when a popup shows up, scrape the Kecamatan’s name close the popup go on to the next Kecamatan, then execute the 6th to 8th instructions above until reaching the last Kecamatan

Okay, let’s dig in.

First step

We’re going to get all Kecamatan areas on the map. If we inspect any area, each Kecamatan’s element is in a <path></path> tag. Such a tag is based on Scalable Vector Graphics (SVG) and inside it, there is a d attribute. Such attribute defines the path of Kecamatan’s area to be drawn so that we can use it to get the coordinates of each Kecamatan’s area. We can use the code snippet below.

Second step

The d attribute consists of the x and y coordinates of each Kecamatan area. The x coordinates are located after an M or L letter, whereas the y coordinates are after a whitespace or before an L or z letter. See the image below to understand this concept better.

The example positions of each X and Y coordinate in a `d` attribute

See the link below for more information about how to utilise the attribute to collect the coordinates.

We can use the code snippet below. It is assumed that you have some basic understanding of how to use regular expression since it is used here to get the coordinates.

Third step