How to run spark batch jobs in AWS EMR using Apache Livy | by Bogdan Cojocar | Towards Data Science

In this article we will discuss about running spark jobs on AWS EMR using a rest interface with the help of Apache Livy.
We will run through the following steps:
creating a simple batch job that reads data from Cassandra and writes the result in parquet in S3
build the jar and store it in S3
submit the job and wait for it to complete via livy
What is Apache Livy?
Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or an RPC client library. Apache Livy also simplifies the interaction between Spark and application servers, thus enabling the use of Spark for interactive web/mobile applications.
Why would you use it?
Apache livy makes our life easier. We don’t need to use EMR steps or to ssh into the cluster and run spark submit. We just use a nice REST interface.
Let’s see it in action.
The batch job will be a scala application.
In our app we first build the spark session and we make sure we can connect to the Cassandra cluster:
val sparkConf = new SparkConf()sparkConf.set("spark.cassandra.connection.host", "YOUR_CASSANDRA_HOST")sparkConf.set("spark.cassandra.connection.port", "YOUR_CASSANDRA_PORT")val spark: SparkSession = SparkSession.builder()      .appName("RunBatchJob")      .config(sparkConf)      .getOrCreate()
We also need to make sure we have access to the S3 bucket where we want to write the data:
spark.sparkContext.hadoopConfiguration.set("fs.s3a.acl.default", "BucketOwnerFullControl")
We can read the input data from the database:
val input = spark.read      .format("org.apache.spark.sql.cassandra")      .options(Map("table" -> "my_table", "keyspace" -> "my_schema"))      .load()
The Cassandra table has a very simple structure:
CREATE TABLE my_schema.my_table (    id1 text PRIMARY KEY,    id2 text,    id3 text,    id4 text);
Finally we simply write the dataframe to a parquet file in S3, while also removing the duplicated rows:
input      .dropDuplicates      .write      .mode("append")      .parquet("s3a://your_bucket/your_preffix/")
Ok, so now we have an app. We need to pack it into a fat jar and copy it to S3. In this way it will be easier to access the jar remotely (from livy).
The dependencies of our app are managed using gradle. At the end of the tutorial I will post the link to the github repo, where you can find the full code and details about the gradle file. I will not into details here, as it’s outside of the scope of this tutorial.
To create the fat jar we need to be inside the root of the app and run:
gradle shadowJar
Once this is generated we can copy it to S3 using the aws cli:
aws s3 cp build/libs/spark_batch_job-1.0-SNAPSHOT-shadow.jar s3://your_bucket/your_prefix/
Once this is done, we can finally start our spark job from using livy.
We will use a simple python script to run our commands. The main function is very simple:
def run_spark_job(master_dns):    response = spark_submit(master_dns)    track_statement_progress(master_dns, response)
It ill first submit the job, and wait for it to complete. The track_statement_progress step is useful in order to detect if our job has run successfully. The master_dns is the address of the EMR cluster. Let’s dive deeper into our individual methods.
The spark_submit function:
host = 'http://' + master_dns + ':8999'data = {'className': "com.app.RunBatchJob", "conf":{"spark.hadoop.fs.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"}, 'file': "s3a://your_bucket/spark_batch_job-1.0-SNAPSHOT-shadow.jar"}headers = {'Content-Type': 'application/json'}response = requests.post(host + '/batches', data=json.dumps(data), headers=headers)
This is just a post done using the requests library. On EMR the livy server runs on port 8999. The data in our request is essentially the parameters we would give to a spark-submit comamnd. We need to use the /batches endpoint. This will tell livy we will submit a batch job.
And the track_statement_progress function. This will poll the livy server every 10 seconds and check if the status of the app is success. If it is then the job has finished successfully:
statement_status = ''host = 'http://' + master_dns + ':8999'session_url = host + response_headers['location'].split('/statements', 1)[0]while statement_status != 'success':        statement_url = host + response_headers['location']        statement_response = requests.get(statement_url, headers={'Content-Type': 'application/json'})        statement_status = statement_response.json()['state']        logging.info('Statement status: ' + statement_status)        lines = requests.get(session_url + '/log', headers={'Content-Type': 'application/json'}).json()['log']        for line in lines:            logging.info(line)        if statement_status == 'dead':            raise ValueError('Exception in the app caused it to be dead: ' + statement_status)        if 'progress' in statement_response.json():            logging.info('Progress: ' + str(statement_response.json()['progress']))        time.sleep(10)
This is a longer function so I will try to explain it step by step. For longer running jobs, livy will change the url, so we need to store the latest one:
statement_url = host + response_headers['location']
Then we get the current status of the cluster and we log it:
statement_response = requests.get(statement_url, headers={'Content-Type': 'application/json'})statement_status = statement_response.json()['state']logging.info('Statement status: ' + statement_status)
We are also getting the logs of the app and displaying them. This is usefull to debug errors:
lines = requests.get(session_url + '/log', headers={'Content-Type': 'application/json'}).json()['log']for line in lines:    logging.info(line)
If we have a severe exception then the app will die. We need to flag that and throw an exception:
if statement_status == 'dead':    raise ValueError('Exception in the app caused it to be dead: ' + statement_status)
And as the final step, we log the progress and wait 10 seconds until we check again the status of the app:
if 'progress' in statement_response.json():    logging.info('Progress: ' + str(statement_response.json()['progress']))time.sleep(10)
And this is all. The code is on github and can be accessed from here.