AWS Glue Studio—No Spark Skills-No Problem
By: Manoj Kukreja
Tags: Machine Learning, Data Science, Artificial Intelligence, AWS, Data

AWS Glue Studio—No Spark Skills-No Problem

Easily create Spark ETL jobs using AWS Glue Studio — no Spark experience required

Image by Gerd Altmann from Pixabay

AWS Glue Studio was launched recently. With AWS Glue Studio you can use a GUI to create, manage and monitor ETL jobs without the need of Spark programming skills. Users may visually create an ETL job by visually defining the source/transform/destination nodes of an ETL job that can perform operations like fetching/saving data, joining datasets, selecting fields, filtering etc. Once a user assembles the various nodes of the ETL job, AWS Glue Studio automatically generates the Spark Code for you.

AWS Glue Studio supports many different types of data sources including:

S3

RDS

Kinesis

Kafka

Let us try to create a simple ETL job. This ETL job will use 3 data sets-Orders, Order Details and Products. The objective is to Join these three data sets, select a few fields, and finally filter orders where the MRSP of the product is greater than $100. Finally, we want to save the results to S3.

When we are done the ETL jobs should visually look like this.

Image by Author

Lets start by downloading the data sets required for this tutorial. Save the data sets in S3.

$ git clone https://github.com/mkukreja1/blogs $ aws s3 mb s3://glue-studio

make_bucket: glue-studio $ aws s3 cp blogs/glue-studio/orders.csv s3://glue-studio/data/orders/orders.csv

upload: blogs/glue-studio/orders.csv to s3://glue-studio/data/orders/orders.csv $ aws s3 cp blogs/glue-studio/orderdetails.csv s3://glue-studio/data/orderdetails/orderdetails.csv

upload: blogs/glue-studio/orderdetails.csv to s3://glue-studio/data/orderdetails/orderdetails.csv $ aws s3 cp blogs/glue-studio/products.csv s3://glue-studio/data/products/products.csv

upload: blogs/glue-studio/products.csv to s3://glue-studio/data/products/products.csv



We will save these files to S3 and catalog them in the orders database using the Glue Crawler.

$ aws glue create-database --database-input '{"Name":"orders"}' $ aws glue create-crawler --cli-input-json '{"Name": "orders","Role": "arn:aws:iam::175908995626:role/glue-role","DatabaseName": "orders","Targets": {"S3Targets": [{"Path": "s3://glue-studio/data/orders/"},{"Path": "s3://glue-studio/data/orders/"}]}}' $ aws glue start-crawler --name orders $ aws glue delete-crawler --name orders $ aws glue create-crawler --cli-input-json '{"Name": "orderdetails","Role": "arn:aws:iam::175908995626:role/glue-role","DatabaseName": "orders","Targets": {"S3Targets": [{"Path": "s3://glue-studio/data/orderdetails/"},{"Path": "s3://glue-studio/data/orderdetails/"}]}}' $ aws glue start-crawler --name orderdetails $ aws glue delete-crawler --name orderdetails $ aws glue create-crawler --cli-input-json '{"Name": "products","Role": "arn:aws:iam::175908995626:role/glue-role","DatabaseName": "orders","Targets": {"S3Targets": [{"Path": "s3://glue-studio/data/products/"},{"Path": "s3://glue-studio/data/products/"}]}}' $ aws glue start-crawler --name products $ aws glue delete-crawler --name products

Using the AWS console open AWS Glue service and click on AWS Glue Studio using the left menu. Make sure you have Blank Graph selected. Click on Create.

Image by Author

Start by creating the first Transform Node-Fetch Orders Data

Image by Author

Make sure that Fetch Orders Data points to the orders table catalogued in Glue previously.

Image by Author

Using the same principles as above create the Transform Node-Fetch OrderDetails Data as well as Fetch Products Data

Now we will create a Transform Node that will join Fetch Orders Data to Fetch OrderDetails Data.

Image by Author

Notice how the joining condition is defined between the two tables as below.

Using the same principles create a Transform Node that will join Join Orders to Fetch Products Data.

Image by Author

Since we want to select a subset of columns from the three table we can use the Select Fields Node

Image by Author

Notice how you can check boxes for fileds that should be included in the final result set.

Image by Author

Now we would like to filter the products whose MRSP is greate than $100. This can be achived by creating a Filter Products MRSP>100 Node as below.

Image by Author

Notice how one or more filter condition can be defined.

Image by Author

Finally, we want to save the result table to S3 in Parquet format. For tis we create a Destination Node-Save Results

Image by Author

Image by Author

Use the Save button to save your ETL job. At this time you should be able to see that AWS Glue Studio has automatically generated the Spark code for you. Click on the Script menu to view the generated code.

We are all set. Lets run the job using the Run button on top right. Click on Run Details should show you the status of the running job. Once the job status changes to Succeeded you can go to S3 to check the final results of the job.

Image by Author

At this point there should be many Parquet files produced in the results folder.

Image by Author

You can check the cotenets of the file using the Apache Parquet Viewer.

Image by Author

I hope this article was helpful. AWS Glue Studio is covered as part of the AWS Big Data Analytics course offered by Datafence Cloud Academy. The course is taught online by myself on weekends.