Extract, Transform, Load (ETL) — AWS Glue | by Furqan Butt | Towards Data Science

AWS Glue is a fully managed, server-less ETL service which can be used to prepare and load data for data analytics purposes. The service can be used to catalog data, clean it, enrich it, and move it reliably between different data stores. In this article I will be explaining how we can use AWS Glue to perform ETL operations in Spark on the Novel Corona Virus Dataset.
The article will cover the following topics:
Glue Components.
Complete tutorial to author a Glue Spark Job.
Extracting data from AWS S3.
Transforming Data using Spark.
Storing Transformed data back to S3 in parquet format.
In a nutshell, AWS Glue has following important components:
Data Source and Data Target: the data store that is provided as input, from where data is loaded for ETL is called the data source and the data store where the transformed data is stored is the data target.
Data Catalog: Data Catalog is AWS Glue’s central metadata repository that is shared across all the services in a region. This catalog has table definitions, job definitions, and other control information to manage your AWS Glue environment.
Crawlers and Classifiers: A crawler is a program that retrieves the schema of data from the data store(s3). Crawler uses custom or built-in classifiers to identify data formats and populates the metadata tables in the catalog.
Database and Table: Each successful crawler run populates a table of a database in the data catalog. A database in the catalog is a set of associated tables. Each table only has metadata information of the data such as names of columns, data type definitions, partition information and the actual data remains in the data store. One or more tables in the database are used by the source and target in an ETL job run.
Job and Triggers: It is the actual business logic to carry out the ETL task. A job is composed of a transformation script, data sources and data targets. We can define our Jobs either in python or pyspark. Job runs are initiated by triggers that can be scheduled or triggered by events.
I have left a few components as they are not under the scope of this article. For a detailed study of AWS Glue you can visit the official developer guide.
Now that we have an understanding of what are the different components of Glue we can now jump into how to author Glue Jobs in AWS and perform the actual extract, transform and load (ETL) operations.
The dataset is obtained from Kaggle Datasets. The version I’m using was last updated on May 02, 2020. Main file in this dataset is covid_19_data.csv The detailed descriptions of the dataset are below.
Sno — Serial number
ObservationDate — Date of the observation in MM/DD/YYYY
Province/State — Province or state of the observation (Could be empty when missing)
Country/Region — Country of observation
Last Update — Time in UTC at which the row is updated for the given province or country. (Not standardized and so please clean before using it)
Confirmed — Cumulative number of confirmed cases till that date
Deaths — Cumulative number of of deaths till that date
Recovered — Cumulative number of recovered cases till that date
As a first step of developing our end to end ETL job we will first setup our data stores. Goto yout s3 console and create a bucket there. We’ll upload the dataset file in the bucket with the following partition scheme:
s3://bucket-name/dataset/year=<year>/month=<month>/day=<day>/hour=<hour>/
Right now we are dealing with a single file so you can create partitions and upload file manually but if working with a large number of files you can use my FTP file ingestion code that I explained in my last article to do this work for you. Partitioning data in this way helps in query optimization while working with AWS Athena.
Create a Glue role that will allow Glue to access different AWS resources e.g s3. Goto the IAM console and add a new role and attach the AWSGlueServiceRole policy to this role. This policy contains permissions to access Glue, CloudWatch, EC2, S3, and IAM. For more details about how to setup IAM role for Glue consider the following link.
Perform the following steps to add a crawler:
On the left menu click on databases and add a database.
Now goto crawlers and and a new crawler
Select data store
Provide the s3 bucket path
Select the Glue role
Set frequency to Run on Demand
Select the database
Finally review and click finish
Now your crawler is created. Click on “Run Crawler” to catalog the dataset
Crawler can take sometime to catalog the data. After successful run you must have tables created in your specified database
Now that we have cataloged our dataset we can now move towards adding a Glue Job that will do the ETL work on our dataset.
On the left menu click on “Jobs” and add a new job. Glue can auto generate a python or pyspark script that we can use to perform ETL operations. However, in our case we’ll be providing a new script.
Set the job properties as follows
Leave the following as default
Set the maximum capacity to 2 and Job Timeout to 40 mins. The higher the number of DPUs(maximum capacity) you set the more cost you will incur.
Since we are not connecting to any RDBMS we don’t have to setup any connections. Click “Save Job and edit script”.
We will have the following screen. Copy and Paste the code from my GitHub repository in the editor below and hit save. Now click the run job button.
Depending upon the work, the Job can take sometime to execute (between 15 to 30 mins in this case). As of now, Glue Jobs have a cold start time of at least 10 mins after which the job execution starts.
If the job execution is successful, you will have aggregated results in parquet format in your destination bucket that is specified in the Glue Job.
If you followed all the above mentioned steps then you should have a successful ETL job execution via AWS Glue. In this section, I will dive into the Spark Code that is doing the actual ETL operations.
At the very top we have the necessary Glue and Spark imports.
After the imports we have a few parameters setup. This includes getting job name, setting up spark and glue contexts, initializing job and defining the catalog database and table names and s3 output path.
The “extract” part of an ETL operation does nothing but connecting to some data store and getting the data from there. The extract portion of the code does following:
creates a Glue DynamicFrame using from the glue catalog. The catalog database and table name are already provided.
After creation of DynamicFrame we covert it to Spark dataframe using the toDF() method. Converting to spark dataframe will allow us to use all the spark transformations and actions.
In the “transform” part of an ETL operation we apply different transformations on our data. The transform part of the code does the following:
First we drop the “last update” column(for no particular reason) using the drop() method in spark.
Then drop any row having more then 4 null fields using the dropna() method in spark.
Then we populate the missing values in “province/state” column with a custom value of “na_province_state” using the fillna() method in spark.
Next we perform aggregations on our dataset. 3 different aggregations have been performed. We check which province/state of a country/region had the most cases, most deaths and most recoveries. This was done by grouping the records by Province/State and Country/Region column using the groupBy() method, aggregating with max(Confirmed),max(Deaths) and max(Recovered) columns using the max() method then sorting them in descending order using the orderBy() method.
Finally we convert our dataframe back to Glue DynamicFrame using the fromDF() method to save the results in S3. It takes three parameters the dataframe, glue context and the name of the resulting DynamicFrame.
In the load part of and ETL operation we store the transformed data to some persistent store such as s3. The load part of code does the following:
We save the results to s3 using the from_options() method of DynamicFrame(). This method takes following parameters:
frame: the DynamicFrame we want to write.
connection_type: the target data store we are writing to in this case s3.
connection_options: here we specify the target s3 path and data the format(parquet in this case) to save the data.
transformation_ctx: optional transformation context.
All the 3 aggregated results are saved on the destination paths in parquet format.
In this article we learned how we can use AWS Glue to do ETL operations in Spark. We learned how we can setup data source and data target, creating crawlers to catalog the data on s3 and authoring Glue Spark Job to perform extract, transform and load(ETL) operations.
The complete spark code can be found on my GitHub repository:
github.com
Thank you for reading ❤.