Web Scraping — Make your Own Dataset | by Suraj Gurav | Towards Data Science

Even though numerous resources available to get the required dataset, it is good to have a self-created dataset. In the current scenario of COVID-19, I found interesting data on worldometers.com. It shows a table containing total number of cases, total number of tests performed, total number of closed cases and so on. This table is updated daily.
Just out of curiosity, I wanted this COVID-19 data in my own dataset. With the help of basic python libraries, pandas, numpy and BeautifulSoup I decided to scrape this entire data into my own data set. As I successfully created the dataset, I would like to share my story with you all. Hence, I will walk you through my Jupyter-Notebook.
Web scraping is the process of collecting information from the web pages.
When large amount of data is to be collected manually, huge amount of time and efforts is required. This is intense if the data to be collected is from regularly updated web pages. Hence, it is always better to have a script, which automatically extracts the data from web page and stores the it in required format.
All you needed is a python package to parse HTML data from the website, arrange it in a tabular form and do some data cleaning. There are dozens of packages available to do this job, but my story continues with BeautifulSoup.
BeautifulSoup
It is a python library, which enables us to scrape all the content from a web page. Suppose there is a web page which contains some interesting information but provides no way to download the data directly. BeautifulSoup provides a set of tools to pull this data from the web page and locate the content which is hidden within the HTML structure.
Let us jump into the process of creating own dataset...
Step 1: Install the required packages
You need requests package which allows you to send HTTP requests using Python. If you have it pre-installed, you need to just import it. Otherwise you need to install it using pip installer.
pip install requestspip install beautifulsoup4
I used Python built-in HTML parser to transform the input text data from website into complex parse tree, which can be further processed to get the required data. If you want to use different parser, here you can find a assistance.
Once you install all the packages, import them in your script.
import pandas as pdimport numpy as npimport requestsfrom bs4 import BeautifulSoup
Step 2: BeautifulSoup Constructor
BeautifulSoup constructor takes 2 input arguments
string — it is the markup that you want to parse. This markup is obtained using get () method of python requests package. This get() method returns the requests.Response()object which contains server’s response to the HTTP request. Additionally, text method of this response object gets the content of the response. This entire string is passed as first argument of BeautifulSoup constructor.
parser — it is the method with which you want to parse the markup. You are expected to provide the name of the parser. I used python built-in HTML parser.
website='https://www.worldometers.info/coronavirus/#countries'website_url=requests.get(website).textsoup = BeautifulSoup(website_url,'html.parser')
That’s it.. In 3 lines of code you got all the content from required web page into your notebook.
All the HTML text from Elements tab (marked in red), in developer tools will be stored in BeautifulSoup objectsoup. As seen in Picture 1 above, when you scroll over specific row in Elements tab , it will highlight corresponding cell in actual table on the website.
Step 3: Pandas DataFrame
In order to make a Pandas DataFrame, we need to transform all the text data in object soup in tabular format. In Picture 1 above, I also marked 3 green rectangles. These are the terms which you need to find in soup.
Let us understand briefly, what does these terms mean —
<tbody> : It specifies the table on the web page.
<tr> : It points to the specific row in this table.
<td> : It points to the particular cell in this row.
In python script, you will use methods of object soup to find these term in the parsed text.
my_table = soup.find('tbody')
variable my_table will contain the table on the web page but still in HTML format. It contains the other two terms <tr> and <td> which points to individual rows and cells respectively. In below code, you will transform this my_table into actual table containing the columns and values.
Please note that, below piece of code is written in April 2020. On later date, kindly check the order of the columns on the website and modify the code accordingly.
table_data = []for row in my_table.findAll('tr'):    row_data = []    for cell in row.findAll('td'):        row_data.append(cell.text)    if(len(row_data) > 0):        data_item = {"Country": row_data[0],                     "TotalCases": row_data[1],                     "NewCases": row_data[2],                     "TotalDeaths": row_data[3],                     "NewDeaths": row_data[4],                     "TotalRecovered": row_data[5],                     "ActiveCases": row_data[6],                     "CriticalCases": row_data[7],                     "Totcase1M": row_data[8],                     "Totdeath1M": row_data[9],                     "TotalTests": row_data[10],                     "Tottest1M": row_data[11],        }        table_data.append(data_item)
Now you get the table_data which is actually a table with columns and values. This table will be now transformed into a pandas DataFrame df.
df = pd.DataFrame(table_data)
Note that, all the columns are of object type in this DataFrame. Also, there are some unwanted characters such as /, \n, |, + in these objects. All the commonly used data cleaning processes can now be used on it.
As per the individual requirement, you can extract different columns and rows from the original DataFrame, clean it and transform it as per requirement.
Such collection of data, its assessment, cleaning and transformation is called as Data Wrangling process. Here is a quick read with interesting visualizations about Data Wrangling process.
towardsdatascience.com
Step 4: Export to Excel
The cleaned, transformed final DataFrame can be then exported to excel using pandas.
df.to_excel('Covid19_data.xlsx', index=True)
Summing up, you created your own dataset for COVID-19.
You can also create multiple data sets by scraping different web pages and finally combine all datasets together. Want to know more about combining different datasets ??
Here I have a simple 4-minute read about merging options in Python Pandas. All you need to remember is the last picture in the article below, which explains all types of data sets join operations.
towardsdatascience.com
And such self-created data set can be easily visualized using Matplotlib Python library and visualizations can be customized easily as given in the article below.
medium.com
Through my story...
I walked you through 4 simple steps to scrape a web page into a excel sheet. I demonstrated how a web page can be scraped using easy to use BeautifulSoup package and transform this data into a pandas DataFrame. The complete notebook is available on my Github profile.
Here are few resources, which can help you with this topic
BeautifulSoup
Requests
Pandas
Web scraping
You are always welcome for a feedback with a comment and connect with me on LinkedIn!!!