#2 Data Engineering — PIPELINES. This is the second blog in the series…
By: Sakshi Agarwal
Tags: Data Engineering, Udacity, Data Science, Etl, Pipeline

This is the second blog in the series of posts related to Data Engineering. I am going to write down all the important things that I learn as a part of the Data Scientist Nanodegree Program, Udacity. I have realized it is the best way to test my understanding of the course material and maintain the discipline to study. Please check the other posts as well.

The first thing we will learn as Data Engineer is Data Pipelining.

Pipelines — INTRODUCTION

Pipelining is nothing but moving data from one place to another. There are two major pipelines:

ETL

An ETL pipeline is a specific kind of data pipeline and is very common. ETL stands for Extract, Transform, Load. Imagine that you have a database containing web log data. Each entry contains the IP address of a user, a timestamp, and the link that the user clicked.

What if your company wanted to run an analysis of links clicked by city and by day? You would need another data set that maps an IP address to a city, and you would also need to extract the day from the timestamp. With an ETL pipeline, you could run code once per day that would extract the previous day’s log data, map the IP address to city, aggregate link clicks by city, and then load these results into a new database. That way, a data analyst or scientist would have access to a table of log data by city and day. That is more convenient than always having to run the same complex data transformations on the raw web log data.

Before cloud computing, businesses stored their data on large, expensive, private servers. Running queries on large data sets, like raw web log data, could be expensive both economically and in terms of time. But data analysts might need to query a database multiple times even in the same day; hence, pre-aggregating the data with an ETL pipeline makes sense.

ELT

ELT (Extract, Load, Transform) pipelines have gained traction since the advent of cloud computing. Cloud computing has lowered the cost of storing data and running queries on large, raw data sets. Many of these cloud services, like Amazon Redshift, Google BigQuery, or IBM Db2 can be queried using SQL or a SQL-like language. With these tools, the data gets extracted, then loaded directly, and finally transformed at the end of the pipeline.

However, ETL pipelines are still used even with these cloud tools. Oftentimes, it still makes sense to run ETL pipelines and store data in a more readable or intuitive format. This can help data analysts and scientists work more efficiently as well as help an organization become more data-driven. We will learn about ETL in the following articles.

ETL Pipeline —Brief Introduction

ETL Pipeline Structure (Image from Databricks)

As a data engineer, you extract data from a place and then transform into another format and further load it into storage someplace else.

(Image from Udacity)

Data Scientists start working on the data after the ETL process is completed. However, in small-scale companies, most people do the task of both Data Engineer and Data Scientist. The consequent articles will give insights into everything you will need to get the job done.

Here is the outline we will follow:

Extract data from different sources such as:

CSV files

JSON files

APIs

Transform data

combining data from different sources

data cleaning

data types

parsing dates

file encodings

missing data

duplicate data

dummy variables

remove outliers

scaling features

engineering features

Load

send the transformed data to a database

ETL Pipeline

code an ETL pipeline

Continue looking this space on to get started with ETL Pipelines.