aws Glue :: A nightmare to developers
By: Abhishek Kulkarni
Tags: Spark, AWS, Big Data, Apache Spark, Aws Glue

First thing, I don’t hate aws or aws glue in particular. I think it is a wonderful service offered by amazon to process big data. Then why the headline, well aws has made major tradeoffs while integrating their other big data solutions to work alongside /integrate with “glue” and they are not good for development.

BootTime: Spinning up spark clusters does take some time, but they should be consistant on an average. Boot time can take anywhere from 1 minute to 15 minutes. Usually this is not a problem in production as we know that we have to wait for maximum of 15 minutes to start the process and processing can take several hours to complete. It can be a huge problem during development. Even to catch a basic error, developer has to wait till job starts running.

Isn’t it developer has to check these “basic” errors before pushing to glue job and expect to work the same way? Not possible for most of the cases specially where GlueContext is been used.

Glue Context: is a wrapper built on Spark that connects glue to other big data services offered by aws (athena/Redshift). This integration is an issue for developers.

GlueContext is not opensource, that can be installed locally to check the services.

Spark has been modified in some cases such as jdbc (database connection to load data from redshift or any similar services) so that will force developer to use Glue Context. Ofcourse there are work arounds, but to figure out if these work arounds actually work, you need to test it in Glue Job.

Things would have been easier if athena is kept as pure as presto and Spark was not modified to integrate to other services.

Why would you have to try these in Job, you can use Zeppelin notebooks?

Zeppelin Notebooks: These notebooks does offer very identical environment to glue. Are very expensive to test out only few parts of the code whereas Jupyter notebooks would have been more than enough. Google’s similar service dataproc and azure databrics offers these at ease.

Monitoring Job Run(boto3): Waiters are calls that can be made to check the status of any long running task. For instance moving a large file from one bucket to another can take some time, there is a waiter available that notifies once the task is completed. For glue there are no registered waiters, application code has to take care of monitoring glue job run (long-poll).

Constraint check: glue has some limitations on invoking glue jobs and these contraints are quite practical and hard to cross. Issue is how easy it is for developer to check these contraints before invoking a new job run?

Lets look at those limitations that are necessary if you are orchestrating the entire service (programmatically invoking glue, calling other things once processed)

Number of concurrent job runs per account: 250

Number of concurrent job runs per job: 1000

Maximum DPUs used by a role at one time: 300

There are no direct api to get count of “concurrent job runs per account” or “Maximum DPUs used by a role at one time” . The only way that is possible is:

get list of all Jobs defined

get list of all job runs for all defined jobs (loop)

filter only running jobs in all job runs, get dpu allocated

sum all the dpu’s collected from previous step

also handle throttle exception when calling api’s

My current solution above mess: