Develop glue jobs locally using Docker containers | by Shubham Jain | Towards Data Science

With the increasing amount of data in the current world, the need for services handling big data is very demanding. When we think about big data, there are very few frameworks which are used by most of the data engineers. Apache Spark is one of them.
Apache Spark is a distributed processing engine which requires a large cluster to execute any kind of analytics or simply perform ETL on the dataset. To use Apache Spark, we need to set up large clusters of our own which are very expensive or we can leverage them on the cloud. Here, we will be discussing the services provided by AWS. So to process large dataset and to perform analytics on them using spark, AWS provides two major services
AWS EMR (Elastic Map Reduce)AWS Glue
AWS EMR (Elastic Map Reduce)
AWS Glue
AWS EMR provides us cluster to perform our processing and are very expensive, so we need to be careful to identify for how long we require clusters and how to optimize our jobs properly. We will not be covering EMR in this post as it is itself a very large topic to discuss.
AWS Glue is a fully managed ETL service provided by amazon web services for handling large amount of data. The best part of AWS Glue is it comes under the AWS serverless umbrella where we need not worry about managing all those clusters and the cost associated with it. In serverless paradigm, we pay for what we use, so if our job is using only 25 DPU for processing our data and it runs for 20 minutes then we end up paying only the cost for leveraging 25 DPU’s for 20 minutes and not a single penny extra.
AWS Glue is in utmost demand nowadays for creating python or scala based spark processing jobs. To develop jobs on AWS glue, we have three options. Let’s discuss them one by one.
Create Job Directly in glue editor and run the job:
Create Job Directly in glue editor and run the job:
If you are a spark developer and not so familiar with glue libraries and its diagram generation, avoid using this editor at all cost. This is for those who are very sure what they are writing and knew from their heart that their job will run in the first run itself. This is not at all recommended and can only be considered when we need to make a few changes in the existing job. Using the job editor directly is good when you are using glue libraries in your code as it generates the lineage diagram as you write the code. Before opting for this option I would recommend going through the glue libraries here.
2. Create a Glue Dev endpoint:
You can set up a glue dev endpoint and link it to either zeppelin notebooks or to your pycharm IDE or you can even connect local jupyter notebooks. This is a better option if you want to test your jobs while you are developing them. To setup Glue Dev endpoint you can follow official AWS documentation here. The only problem with this glue dev endpoint is its cost. You will be charged for the time it is running, so it basically spins up a EMR cluster with your desired capacity and glue libraries installed on the cluster. This is one of the best option if cost is not a problem with your development and can be considered.
3. Use AWS Glue libraries and run them on Docker container locally
This is by far the best option considering the development of the jobs and testing the jobs on relatively small datasets and once the job is ready running them using the glue job console itself. With this option, you get the flexibility of using glue libraries either on your machine or you can spin up EC2 instances according to your dataset size and launch the docker container on these EC2, as this will be a relatively cheaper option and the most suitable one to run your jobs.
To set up AWS glue locally, you can follow AWS documentation here. But this seems to be not working and we will be discussing it in detail. So I have created a Docker image and will explain what we are doing.
I am taking centos as base image then setting these env variables we will require this to download the libraries later on
ENV MAVEN=https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gzENV SPARK=https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-1.0/spark-2.4.3-bin-hadoop2.8.tgzENV GLUE=https://github.com/awslabs/aws-glue-libs.git
Then we will create a directory glue and install required libraries like Java, Python, wget, tar and python packages and set our working directory to glue
RUN mkdir glueRUN yum install -y python3 java-1.8.0-openjdk java-1.8.0-openjdk-devel tar git wget zipRUN ln -s /usr/bin/python3 /usr/bin/pythonRUN ln -s /usr/bin/pip3 /usr/bin/pipRUN pip install pandasRUN pip install boto3RUN pip install pyntWORKDIR ./glue
Now in this directory will download maven, spark and glue libs and unzip them. Then we will be updating the path variable and adding SPARK_HOME, JAVA_HOME, MAVEN_HOME and GLUE_HOME variables to it.
#To get the latest aws librariesRUN git clone -b glue-1.0 $GLUE#To get latest spark versionRUN wget $SPARK#To install Maven for dependenciesRUN wget $MAVENRUN tar zxfv apache-maven-3.6.0-bin.tar.gzRUN tar zxfv spark-2.4.3-bin-hadoop2.8.tgz#Removing zip files inorder to reduce the final docker image sizeRUN rm spark-2.4.3-bin-hadoop2.8.tgzRUN rm apache-maven-3.6.0-bin.tar.gz#Setting up env variablesRUN mv $(rpm -q -l java-1.8.0-openjdk-devel | grep "/bin$" | rev | cut -d"/" -f2- |rev) /usr/lib/jvm/jdkENV SPARK_HOME /glue/spark-2.4.3-bin-spark-2.4.3-bin-hadoop2.8ENV MAVEN_HOME /glue/apache-maven-3.6.0ENV JAVA_HOME /usr/lib/jvm/jdkENV GLUE_HOME /glue/aws-glue-libsENV PATH $PATH:$MAVEN_HOME/bin:$SPARK_HOME/bin:$JAVA_HOME/bin:$GLUE_HOME/bin
Here is an important known issue where will be removing some of the libraries https://github.com/awslabs/aws-glue-libs/issues/25.
RUN sed -i '/mvn -f/a rm /glue/aws-glue-libs/jarsv1/netty-*' /glue/aws-glue-libs/bin/glue-setup.shRUN sed -i '/mvn -f/a rm /glue/aws-glue-libs/jarsv1/javax.servlet-3.*' /glue/aws-glue-libs/bin/glue-setup.sh
Now we will compile all the dependencies using maven. This shell script run the maven build command and gets all the required dependencies.
We are running this once so that our docker image will contain the libraries beforehand and will save time to run spark jobs later.
RUN sh /glue/aws-glue-libs/bin/glue-setup.sh
At last we will be cleaning all the tmp directories and using bash as our entrypoint to our container.
RUN yum clean allRUN rm -rf /var/cache/yumCMD ["bash"]
Now to use the docker file and create your container in order to start developing your glue spark jobs follow along.
To build the image out of this Dockerfile run the following command:
docker build -t jnshubham/glue_etl_local .
To use the existing image for starting your container, just pull the image from docker hub and run the commands provided to submit jobs
To pull the image run the following command:
docker pull jnshubham/glue_etl_local:latest
Check downloaded image by running
docker images
To run the container and get into repl shell:
docker run jnshubham/glue_etl_local "gluepyspark"
To get into the terminal and submit a spark job run
docker run -it jnshubham/glue_etl_localgluesparksubmit script.py --JOB_NAME script
This way you will be able to test your job on your local machine or you can also use this on any EC2 instance based on your data size.
For more information, visit my GitHub or dockerhub.