Building a ETL pipeline
By: Anthony Inthavong
Tags: Python, Request, Covid 19, Etl, Pandas

using Python, Pandas, and SQLite

What is a ETL data pipeline?

Typical ETL data Pipeline/Process

An ETL (Data Extraction, Transformation, Loading) pipeline is a set of processes used to Extract, Transform, and Load data from a source to a target.

The source of the data can be from one or many sources, such as from an API call, CSV files, information within a database, and many more.

We take these sources of information then transform it in a way where it can be used immediately by another client, user or developer within some target storage.

In practice, an ETL pipeline is run infrequently. Generally there are large amounts of information that is unfiltered and difficult to process and clean, often taking large amounts of time and resources to transform.

Why do we need an ETL pipeline?

Imagine we had a collection of data that we wanted to use. But the problem is this data is uncleaned, missing information, and inconsistent. One solution is to have a program clean and transform this information such that

There is no missing information Data is consistent Data is fast to load into another program

There is so much raw, unfiltered data in today’s industry with smart devices, online communities, and E-Commerce. But most of it is wasted because it is to hard to interpret because of how messy it is. ETL pipelines exist to combat this by automating data collection and transformation so it can be used by analysts for business insights.

The ETL pipeline

There are many different tools and technologies we can used to build our data pipeline. This is completely dependent on how our data is collected. In our example we will be collecting raw data from John Hopkins University’s GitHub. Specifically we will be looking at the daily reports. The data spans from January 22, 2020 to December 16, 2020 as of the writing of this article.

Our tech stack will include Python using the libraries Requests and Pandas. We will also be using SQLite to store this information.

1. Extract

Before we can do any sort of data transformation, we need to have data! The data will be collected from John Hopkins University’s Github. We are specifically interested in the daily reports. From Github’s API documentation, we can send a GET request to this endpoint to receive a JSON response back telling us the urls for each csv file so we can read directly.

OWNER is owner of GitHub repository

REPO is name of repository

PATH is specific path to our folder we want to examine relative to REPO

URL is api endpoint used to send GET request

Using Python’s Requests library we will interpret the JSON response and examine the information we need. For clarity, the image below is the response returned using a program called Postman. A similar response will be returned using Python’s Request library.

JSON response from GitHub API

What is returned is an array of object descriptions for every file in the github repo. We are interested in the ‘download_url’. We will use this ‘download_url’ to download this data directly using Pandas then transform it as we see fit.

The script below collects all the ‘download_url’’s into a Python List Object

2. Transform

We now have a list of direct links to our csv files! We can read these urls directly using pandas.read_csv(url).

Taking a look at the information, we are interested in looking at specific values. If these values are not within the DataFrame, we will replace/add these columns with Empty values (such as numpy.nan).

Some labels are inconsistent. They define the same type of data, but they are named differently. For example, ‘Last Update’ is also called ‘Last_Update’. We will also redefine these labels for consistency. This is a very basic ETL pipeline so we will only consider a small number of labels. These labels will be

labels = ['Province_State', 'Country_Region', 'Last_Update', 'Confirmed', 'Deaths', 'Recovered']



Transformation code

The code above transform the already collected data as a Pandas DataFrame add makes the DataFrames consistent by adding and relabelling columns. With our data cleaned, filtered and processed, the next step is to load the data into a database.

3. Load

We can directly load our pandas DataFrame into a SQL database using pandas.DataFrame.to_sql.

If the table within the database has already created, replace it with a new one. This is a very bad way to do this in production. But for the simplicity of this exercise it suffices.

The code below describes how to connect to a SQLite database using sqlite3.

Conclusion

We described how beneficial the ETL pipeline is in today’s data focused industry and society.

We explained how the ETL process can vary depending on the data being collected, the type of data, and how much data there is.

We implemented our own basic ETL pipeline using Python, Pandas, Requests, and SQLite and interacted with GitHub’s API.

The full code can be found here