Automated Web Scraping Using Cron | Towards Data Science

Sometimes, we want to do some task daily. It seems boring if you do it by yourself, and you waste your time just for doing it. Automation is what we need. By scheduling it at a given time, we can save our time, and let the computer do the task itself.
In this article, I want to show you how to build a COVID-19 dataset for Riau Province, Indonesia using Python to retrieve the dataset and also Cron to schedule the task. If you want to know more, you can check about it on my GitHub repository here.
Riau is a province in Indonesia. Just like other places, this place also fights COVID-19. They have the source of information which is corona.riau.go.id. The front page looks like this,
On the website, it shows the number of cases that exist at the province and regional level. Although it is up to date, they do not show the historical data of it. Just like these pictures below,
Because of that problem, I propose a web scraping technique to record the data and save them as .csv format. After this, I will show you step-by-step on how I scrape the website, and how to automate those task at a given schedule.
The first thing that I do is to scrape the website. To scrape it, I use the bs4 library to extract the text from it. For this case, There are some problems. First, the table that I’ve shown before is from another website, and it’s just a frame at the web. Therefore, we have to get the source of it.
On Microsoft Edge browser, We can get the source of the frame for each table by right-clicking on it, and select view frame source. For the province-level data, the source comes from https://covid19.riau.go.id/webster. For the city or regency level, the source comes from https://covid19.riau.go.id/pantauan_data_kasus. The next step is to scrape each website to get the data that we need.
First, we will extract the data from https://covid19.riau.go.id/webster. Retrieving the data is not so difficult because we only have to inspect which tag belongs to the number. The data that we want to extract is just the numbers of each column, so we can make the column’s name using a list.
Based on the picture above, the <td> tag contains the numbers. The <td> tag is encapsulated by the <table> tag. Therefore, we have to extract all of the <table> tags, and after that, we extract the <td> tag content for each <tbody> tag. We use looping to extract the data, and we put it to a variable which has a dictionary structure to it, and after that, we write it to CSV file. The code looks like this,
If we display the .csv file, it looks like this,
After that, we will extract the city regency level dataset.
For the dataset, we will scrape from https://covid19.riau.go.id/pantauan_data_kasus. Based on the picture above, we only have to retrieve all of the <tr> tag first. After that, we can retrieve the <td> tag for each of the <tr> tag. The process to become a .csv file is same as the previous one, and the code looks like this,
After we run the code, it will create a .csv file, and it appends below the existing line. Here is the preview of the .csv file,
After we’ve created the dataset, we can focus on how to update the dataset daily. To achieve this, we can schedule the task using Cron on Linux. In this case, I will use the Raspberry Pi Zero W as my server to scrape the dataset daily, and the operating system that I use is Raspbian.
You’ve noticed that I’ve made the script on Windows. Therefore, I’ve made a repository of it on GitHub, so it’s become accessible from any device. If you want to know the code, you can see it on my repository here. I’ve already cloned the repository on my Pi, so we can straight to schedule the task.
Before we create the schedule command, we have to make a file with .sh format to put the script that we want to run. The script that I’ve made looks like this,
The first line is a shebang declarative that tells the system to use the bash interpreter. The line below is the python script that we want to run. After we make this, we can create our schedule command using Cron.
To do this, we can open the terminal first. Then, type crontab -e, and the terminal will open a Nano text editor (you change it if you want to use vim). At first, we will see some note of the crontab file like this,
We can write our schedule task below of it. The structure of it looks like this,
59 23 * * * /bin/bash ~/projects/covid-riau-tracker/script.sh
Well, it seems weird at first, but let me explain. The first five fields describe the schedule of our task. In order, each of them describes the minute (0–60), the hour (0–24), the day of month (1–31), the month (1–12), and the day of week (1–7). After those fields, we can define the command that we want to run.
Because I’ve written the script, we can run it using bash. If you look in detail, I write the absolute path of it. It’s because the Cron doesn’t have the same path as our file. That’s why I write the absolute path to access the file.
After we write the task, we can save it by using CTRL + O, and we can exit to the terminal by using CTRL + X. That’s it, we’ve created an automation to the web scraping task.
In conclusion, web scraping is a useful method to retrieve the data that we want if it doesn’t exist in friendly-format like .csv format. Also, we can do this without running it by yourself, and let the Cron to do that.
Although it is already automated, there is a problem. It’s how to update the dataset, and then push it to GitHub. Until now, I still cannot push the result to the GitHub, and I’m struggling with it. So, if you know how to do that in Cron, leave comments down below.
I think that’s what I can share to you right now. I hope that useful to you on how the web scraping is really useful to retrieve the data that we want, and also how to automate it. Thank you.