Harness the Power of Data With AWS Glue
By: Irin George
Tags: Big Data Analytics, Aws Glue, AWS, Big Data, Cloud Computing

As the world becomes more digital, the amount of data created and collected continuously grows and accelerates. Analysis of this constantly growing data becomes a challenge with traditional analytical tools. Innovation is required to fill the gap between generated data and data that can be analysed effectively.

Big data tools and technologies offer various opportunities to analyse data efficiently which makes us to better understand customer preferences, gain new insights which will give us a better position in the marketplace, and grow our business. In the last few years, the complex architecture of data management which developed from the traditional data warehousing model addresses requirements, such as real-time and batch processing, structured and unstructured, data and so on.

AWS provides various services which help us to build solutions in a secure way and also to scale up accordingly. AWS provides the infrastructure and tools to work with our next big data project which requires real-time streaming or batch data processing. It also provides various analytical solutions which can handle this vast amount of data that is growing and provides insight that helps our business.

WHAT IS AWS GLUE?

AWS Glue is an Extract, Transform, and Load service. What this means is that we can categorize, clean, and enrich our data from various sources and output the data into other sources. AWS Glue has the capability to store schema information from various data sources using AWS Glue Crawler in the central metadata repository and also generate python or scala code for transforming and loading data. It also has a flexible scheduler that handles dependency resolution, job monitoring, etc. Since AWS Glue is serverless there is no need for infrastructure set up to manage it.

The benefits of using AWS Glue are:

Faster Data Integration: Different groups in an organisation can use AWS Glue for data integration which includes cleaning, extracting, loading, etc. This way we can reduce the time taken for data integration. Automate Data Integration at scale: Glue automates most of the tasks which are required in data integration. The AWS Glue Crawler automatically detects the data formats and schema information from various sources and store this information in Glue Data Catalog. It even generates code to run transformation and loading processes. No Servers to Manage: Glue has no server to maintain, there is no infrastructure to manage.

Let’s dig a bit deeper and think about how AWS Glue can be used in Big Data Analytics.

AWS Glue Concepts

The below diagram shows the AWS Glue environment.

We define a Glue job to perform ETL in our data. We typically perform the following actions:

For datastore sources, we define a crawler to crawl metadata information from various data store and store it in Glue Data Catalog.

Adding to table details, the Glue Data Catalog which is a central metadata repository contains other metadata information which is required to define our ETL jobs. We can use this metadata when we define a job to transform our data.

AWS provides an opportunity to use manually and write the code on our own or it can be pre-generated.

We can run a job on-demand or can set it up to start when a specified trigger occurs. The trigger can be a time-based schedule or an event based on our requirements. When our job runs, a script that Glue has, performs extraction, transformation, and loading data into our data target. The AWS Glue has an Apache Spark environment where the script runs.

AWS Glue Terminology

AWS Glue has several components to create and manage our ETL (Extract, Transform, Load) workflow.

AWS Glue Data Catalog

The metadata store in AWS Glue in which Glue jobs can access the metadata information. It contains information such as tables, jobs, and other control information to manage our AWS Glue environment. Each region has one AWS Glue Data Catalog.

Classifier

It determines the schema of our data. It is available for common file types, such as CSV, JSON, AVRO, XML, and others.

Connection

Connection in AWS Glue Data Catalog is an object which has the properties to connect to different data stores.

Crawler

A program that connects to different data store, source or target, determine the schema for our data, and then creates metadata tables in the AWS Glue Data Catalog.

Database

A set of associated Data Catalog table definitions which is organized into a logical group and only requires a name to create a database.

Data store, data source, data target

A data store is a storage for storing your data. Eg: Amazon S3 buckets and relational databases. A data source can be defined as a data store where we can use it as input for processed or transformed data. A data target is where processed or transformed data writes to.

Development endpoint

An environment that can be used to develop and test AWS Glue ETL scripts.

Dynamic Frame

It is similar to data frame and each record in the dynamic frame is self-describing, designed for schema flexibility with semi-structured data. Each record describes the schema which also contains data that gives us a clear picture of the data. We have the opportunity to use both Apache Spark DataFrames and dynamic frames in our ETL scripts and convert between them in AWS Glue.

Job

Glue job is composed of a transformation script, the data sources, and the data targets. This job runs by a triggered schedule or a triggered event to transform and create the data that we want to have in our output data store.

Notebook server

Notebook server is a web-based environment that we can use to run our PySpark scripts. PySpark is a Python language for ETL programming. A notebook server can be set up on a development endpoint to run PySpark code with AWS Glue extensions.

Script

A Python or Scala code that performs ETL in data.

Table

The metadata information represents our data. Whether our data is in an Amazon Simple Storage Service (Amazon S3) file, an Amazon Relational Database Service (Amazon RDS) table, or another set of data, a table defines the schema of our data. A table consists of all metadata information such as column names, data type, partition, and other metadata about a base dataset which are stored in the AWS Glue Data Catalog. The schema of our data is which is represented in AWS Glue table definition has metadata information and actual data is present in data store. AWS Glue list and store only schema information of our files and relational database tables in the AWS Glue Data Catalog. They are used as sources and targets when we create an ETL job.

Transform

The code logic is used to manipulate our data into a different format.

Trigger

Initiates an ETL job. Triggers can be defined based on a scheduled time or an event based on requirements.

Conclusion

AWS Glue can be an efficient and useful tool that is used for data preparation and ETL jobs. I hope that this brief introduction to Glue allows you to make better business decisions with ease.

Looking for a detailed guide on AWS glue? Please refer: https://docs.aws.amazon.com/glue/latest/dg/glue-dg.pdf

Ankercloud is one of the leading Big Data analytics service providers. We help organizations derive systematic analysis of data, so they can make more informed decisions — thereby driving efficiency and improving organizational agility. If you need any help with Big Data or related services, feel free to contact us — info@ankercloud.com