Advanced monitoring of AWS glue jobs by enabling spark UI | by Shubham Jain | Towards Data Science

Apache spark is currently an indispensable framework when it comes to processing huge datasets. Most of the data engineers and data scientists who are dealing with large amounts of data leverages spark capabilities at some point in time.
When it comes to understanding of spark, 20% comprises of the code syntax, 30% are the data engineering concepts and the rest 50% is the optimization
Code syntax consists of the use of functions and the syntax, data engineering concepts reflects the knowledge of SQL, efficient query building and creating optimized query and optimization includes how well the cluster is managed and running the code with the minimal shuffle, dealing with inevitable small files, optimizing joins, dealing with long-running jobs, optimization of skewed datasets and utilizing the complete cluster.
The syntax definition and data engineering concepts can be learned over time but to optimize the cluster you need to visualize how your cluster is responding to the queries executed over it.
SparkUI is needed to identify the root cause behind long-running jobs, Visualizing DAGs, OOM issues, disk spills, excessive shuffling, skewed datasets, and overall health of the cluster.
To use apache spark we need large clusters but sometimes, managing these clusters becomes a bit of additional overhead. So we started leveraging clusters on the cloud using AWS EMR or GCP Dataproc but again we need to manage these clusters and make the most of them. Now there was a need for something where we can run our spark workloads and pay for only what we use. That’s where AWS glue comes into the picture.
Aws Glue is serverless, therefore we don’t need to manage the clusters or worry about the running cost associated with it. We pay only for the time we utilize the resources and as soon our job is done, the resources are deallocated.
Every good thing comes with a price, so the catch here is we cannot fully configure our glue jobs in terms of optimization. Also, we cannot view the spark UI for the jobs in realtime, instead, we need to run a Spark History server which allows us to see the Spark UI for the glue jobs.
To enable the spark UI we need to follow some steps:
Enable spark UI option in glue jobs.Specify the s3 path where the logs will be generated.Start a Spark History Server using docker and EC2.Access spark UI on the History server.
Enable spark UI option in glue jobs.
Specify the s3 path where the logs will be generated.
Start a Spark History Server using docker and EC2.
Access spark UI on the History server.
Create a new job and in the monitoring section enable the spark UI option and provide an s3 path for logs generation.
We will now be creating a Spark History server from which we can visualize the DAG and task details for the glue jobs.
The Dockerfile for the same is:
I am taking alpine as the base image as I want this docker container to be very lite in size.
Installing Java 8, Maven and bash shell
RUN apk add --no-cache openjdk8RUN apk add --no-cache mavenRUN apk add --no-cache bash
Fetching spark 2.4 and unzipping as this contains the class for enabling the history server
RUN wget https://archive.apache.org/dist/spark/spark-2.4.3/spark-2.4.3-bin-without-hadoop.tgzRUN mv spark-2.4.3-bin-without-hadoop.tgz sparkRUN tar -zxf sparkRUN rm -rf sparkRUN mv spark-2.4.3-bin-without-hadoop/ spark/
Copy pom.xml to the container for installing the dependency and build it using maven
COPY pom.xml .RUN mvn dependency:copy-dependencies -DoutputDirectory = /sparkui/spark/jars/
Removing maven as the build is now successful and also removing the conflicting jars
RUN apk del mavenRUN rm /sparkui/spark/jars/servlet-api-2.5.jar && \    rm /sparkui/spark/jars/jsr305-1.3.9.jar && \    rm /sparkui/spark/jars/jersey-*-1.9.jar
Adding the port number to enable the history server, you can change it according to your requirement. We need to add the property to the spark-defaults.conf file.
You can also change the port number here.
RUN echo $’\n\spark.eventLog.enabled true\n\spark.history.ui.port 18080\n\’ > /sparkui/spark/conf/spark-defaults.conf 
Adding the entrypoint to the docker container, this entrypoint will enable the spark history server.
ENTRYPOINT [“/sparkui/spark/bin/spark-class”, “org.apache.spark.deploy.history.HistoryServer”]
Now build and run the docker container from this file.
To build the image out of this Dockerfile run the following command:
docker build -t jnshubham/glue_sparkui .
You can pull the pre-build image available on the docker hub also.
To pull the image run the following command:
docker pull jnshubham/glue_sparkui:latest
Check downloaded image by running
docker images
To run the container and start the history server run the following commands:
To run the container with Access key and secret key
$ LOG_DIR="s3a://path_to_eventlog/"$ AWS_ACCESS_KEY_ID="AKIAxxxxxxxxxxxx"$ AWS_SECRET_ACCESS_KEY="yyyyyyyyyyyyyyy"$ docker run -itd -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$LOG_DIR -Dspark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID -Dspark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY" -p 18080:18080 jnshubham/glue_sparkui
To run the container with the IAM role configured to your EC2 for accessing s3 bucket
$ export LOG_DIR=s3a://bucket_name/logs_path/$ docker run -itd -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$LOG_DIR" -p 18080:18080 jnshubham/glue_sparkui
To access the history server enable the port 18080 in security groups inbound rule to allow traffic.
Once you have some successful logs open the ip:18080 to open the history server which will show the spark ui for the glue jobs:
You can now visualize the DAGs generated and the query performance. You can also track the time taken by each task and then work on the optimization of the same.
Hope you all liked the article.
For more information, visit my GitHub or dockerhub.
Some rights reserved