Running Spark Application in the EMR Cluster through AWS Lambda Function
By: Ankita Kundra
Tags: Emr, AWS, AWS Lambda, S3

Running Spark Application in the EMR Cluster through AWS Lambda Function Ankita Kundra Follow Sep 4 · 7 min read

This post gives you a quick walkthrough on AWS Lambda Functions and running Apache Spark in the EMR cluster through the Lambda function. It also explains how to trigger the function using other Amazon Services like S3.

What is AWS Lambda?

AWS Lambda is one of the ingredients in Amazon’s overall serverless computing paradigm and it allows you to run code without thinking about the servers. Serverless computing is a hot trend in the Software architecture world. It enables developers to build applications faster by eliminating the need to manage infrastructures. With serverless applications, the cloud service provider automatically provisions, scales, and manages the infrastructures required to run the code.

It abstracts away all components that you would normally require including servers, platforms, and virtual machines so that you can just focus on writing the code.

Since you don’t have to worry about any of those other things, the time to production and deployment is very low. Another great benefit of the Lambda function is that you only pay for the compute time that you consume. This means that you are being charged only for the time taken by your code to execute.

This is in contrast to any other traditional model where you pay for servers, updates, and maintenances.

The AWS Lambda free usage tier includes 1M free requests per month and 400,000 GB-seconds of compute time per month. To know about the pricing details, please refer to the AWS documentation: https://aws.amazon.com/lambda/pricing/

You do need an AWS account to go through the exercise below and if you don’t have one just head over to https://aws.amazon.com/console/. If you are a student, you can benefit through the no-cost AWS Educate Program. I would suggest you sign up for a new account and get $75 as AWS credits. I won’t walk through every step of the signup process since its pretty self explanatory.

What is Apache Spark?

Apache Spark is a distributed data processing framework and programming model that helps you do machine learning, stream processing, or graph analytics. It is an open-source, distributed processing system that can quickly perform processing tasks on very large data sets. It is often compared to Apache Hadoop, and specifically to MapReduce, Hadoop’s native data-processing component. The difference between spark and MapReduce is that Spark actively caches data in-memory and has an optimized engine which results in dramatically faster processing speed.

To know more about Apache Spark, you can refer to these links:

https://spark.apache.org/docs/latest/

https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark.html

In this article, I would go through the following:

Creating Lambda functions in Python Integrating it with other AWS services such as S3 Running a Spark job as a Step Function in EMR cluster

I assume that you have already set AWS CLI in your local system. If not, you can quickly go through this tutorial https://cloudacademy.com/blog/how-to-use-aws-cli/ to set it up.

I have tried to run most of the steps through CLI so that we get to know what's happening behind the picture.

1. Creating an S3 bucket

Create an s3 bucket that will be used to upload the data and the Spark code. Get the Arn for the bucket created. To find the ARN for an S3 bucket, you can look at the Amazon S3 console Bucket Policy or CORS configuration permissions pages.

aws s3api create-bucket --bucket <bucket-name> --region us-east-1

2. Creating an IAM Role and Policies

An IAM role is an IAM entity that defines a set of permissions for making AWS service requests. IAM policy is an object in AWS that, when associated with an identity or resource, defines their permissions.

2.1. Creating an IAM policy with full access to the EMR cluster.

Create a file in your local system containing the below policy in JSON format. e.g policy

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "VisualEditor0",

"Effect": "Allow",

"Action": "elasticmapreduce:*",

"Resource": "*"

}

]

}

Run the command to create a policy:

aws iam create-policy --policy-name <policy-name> --policy-document file://<filename>

A similar output will be printed to the console like below:

{

"Policy": {

"PolicyName": "emr-full-access",

"PolicyId": "ANPAQ6X4YKPB7BY",

"Arn": "arn:aws:iam::123456789012:policy/emr-full-access",

"Path": "/",

"DefaultVersionId": "v1",

"AttachmentCount": 0,

"PermissionsBoundaryUsageCount": 0,

"IsAttachable": true,

"CreateDate": "2020-09-04T03:20:42Z",

"UpdateDate": "2020-09-04T03:20:42Z"

}

}

Note down the ARN (highlighted in bold )created which will be used later.

We will be creating an IAM role and attaching the necessary permissions.

2.2. Creating an IAM role

We create an IAM role with the below trust policy. An IAM role has two main parts:

Permission Policy which describes the permission of the role Trust Policy which describes who can assume the role

Create a file containing the trust policy in JSON format. e.g. trust-policy.json

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"Service": "lambda.amazonaws.com"

},

"Action": "sts:AssumeRole"

}

]

}

Run the command to create the role:

aws iam create-role --role-name <role-name> --assume-role-policy-document file://<file-name.json>

Note down the Arn value which will be printed in the console

We need ARN for another policy AWSLambdaExecute which is already defined in the IAM policies. AWSLambdaExecute policy sets the necessary permissions for the Lambda function.

Run the below command to get the Arn value for a given policy

aws iam list-policies --query 'Policies[?PolicyName==`emr-full`].Arn' --output text

2.3. Attaching the 2 policies to the role created above

aws iam attach-role-policy --role-name S3-Lambda-Emr --policy-arn "arn:aws:iam::aws:policy/AWSLambdaExecute" aws iam attach-role-policy --role-name S3-Lambda-Emr --policy-arn "arn:aws:iam::123456789012:policy/emr-full-policy"

Note: Replace the Arn account value with your account number.

Make sure to verify the role/policies that we created by going through IAM (Identity and Access Management) in the AWS console.

3. Creating the Lambda Function

We create the below function in the AWS Lambda. After the event is triggered, it goes through the list of EMR clusters and picks the first waiting/running cluster and then submits a spark job as a step function.

Zip the above python file and run the below command to create the lambda function from AWS CLI.

aws lambda create-function --function-name FileWatcher-Spark \

--zip-file fileb://lambda-function.zip --handler lambda-function.lambda_handler --runtime python3.7 --role arn:aws:iam::123456789012:role/S3-Lambda-Emr

Replace the zip file name, handler name(a method that processes your event). In my case, it is lambda-function.lambda_handler (python-file-name.method-name).

Also, replace the Arn value of the role that was created above.

Once it is created, you can go through the Lambda AWS console to check whether the function got created.

Once we have the function ready, its time to add permission to the function to access the source bucket.

aws lambda add-permission --function-name <FileWatcher-Spark> --principal s3.amazonaws.com \

--statement-id s3invoke --action "lambda:InvokeFunction" \

--source-arn <bucket-arn-value>\

--source-account <123456789012>

Replace the source account with your account value. The account can be easily found in the AWS console or through AWS CLI.

aws sts get-caller-identity

Now its time to add a trigger for the s3 bucket.

Create another file for the bucket notification configuration.eg. notification.json.

{

"LambdaFunctionConfigurations": [

{

"LambdaFunctionArn": "arn:aws:lambda:us-east-1:065987431742:function:FileWatcher-Spark",

"Events": ["s3:ObjectCreated:Put"],

"Filter": {

"Key": {

"FilterRules": [

{

"Name": "suffix",

"Value": ".csv"

}

]

}

}

}]

}

We are using S3ObjectCreated:Put event to trigger the lambda function

aws s3api put-bucket-notification-configuration --bucket lambda-emr-exercise --notification-configuration file://notification.json

Verify that trigger is added to the lambda function in the console

Create a sample word count program in Spark and place the file in the s3 bucket location. Ensure to upload the code in the same folder as provided in the lambda function.

import sys

from pyspark import SparkContext, SparkConf

if __name__ == "__main__":

conf = SparkConf().setAppName("Spark Count")

sc = SparkContext(conf=conf)

input_file = sys.argv[1]

output_file = sys.argv[2]

readTextFile= sc.textFile(input_file).flatMap(lambda line: line.split(" ")) wordCount = readTextFile.map(lambda word: (word, 1)).reduceByKey(lambda v1,v2:v1 +v2) wordCount.coalesce(1).saveAsTextFile(output_file)

Start an EMR cluster with a version greater than emr-5.30.1.

Now, its time to trigger the function.

aws s3api put-object --bucket <bucket-name> --key data/test.csv --body test.csv

Spark job will be triggered immediately and will be added as a step function within the EMR cluster as below:

Summary

This post has provided an introduction to the AWS Lambda function which is used to trigger Spark Application in the EMR cluster. The above functionality is a subset of many data processing jobs ran across multiple businesses. Data pipeline has become an absolute necessity and a core component for today’s data-driven enterprises. The article covers a data pipeline that can be easily implemented to run processing tasks on any cloud platform. Similar to AWS, GCP provides services like Google Cloud Function and Cloud DataProc that can be used to execute a similar pipeline.