Package: cdk-emrserverless-with-delta-lake
cdk-emrserverless-with-delta-lakenpm (JS/TS)PyPI (Python)Maven (Java)GoNuGetLinkLinkLinkLinkLinkThis constrcut builds an EMR studio, a cluster template for the EMR Studio, and an EMR Serverless application. 2 S3 buckets will be created, one is for the EMR Studio workspace and the other one is for EMR Serverless applications. Besides, the VPC and the subnets for the EMR Studio will be tagged{"Key": "for-use-with-amazon-emr-managed-policies", "Value": "true"}via a custom resource. This is necessary for theservice roleof EMR Studio.
This construct is for analysts, data engineers, and anyone who wants to know how to processDelta Lake datawith EMR serverless.They build the construct viacdkv2and build a serverless job within the EMR application generated by the construct via AWS CLI within few minutes. After the EMR serverless job is finished, they can then check the processed result done by the EMR serverless job on an EMR notebook through the cluster template.TOCRequirementsBefore deploymentMinimal content for deploymentAfter deploymentCreate an EMR Serverless applicationCheck the executing jobCheck results from an EMR notebook via cluster templateFun factsFuture workRequirementsYour current identity has theAdministratorAccesspower.An IAM usernamedAdministratorwith theAdministratorAccesspower.This is related to the Portfolio of AWS Service Catalog created by the construct, which is required forEMR cluster tempaltes.You can choose whatsoever identity you wish to associate with the Product in the Porfolio for creating an EMR cluster via cluster tempalte. CheckserviceCatalogPropsin theEmrServerlessconstruct for detail, otherwise, the IAM user mentioned above will be chosen to set up with the Product.Before deploymentYou might want to execute the following command.PROFILE_NAME="scott.hsieh"# If you only have one credentials on your local machine, just ignore `--profile`, buddy.cdkbootstrapaws://${AWS_ACCOUNT_ID}/${AWS_REGION}--profile${PROFILE_NAME}Minimal content for deployment#!/usr/bin/env nodeimport*ascdkfrom'aws-cdk-lib';import{Construct}from'constructs';import{EmrServerless}from'cdk-emrserverless-with-delta-lake';classTypescriptStackextendscdk.Stack{constructor(scope:Construct,id:string,props?:cdk.StackProps){super(scope,id,props);newEmrServerless(this,'EmrServerless');}}constapp=newcdk.App();newTypescriptStack(app,'TypescriptStack',{stackName:'emr-studio',env:{region:process.env.CDK_DEFAULT_REGION,account:process.env.CDK_DEFAULT_ACCOUNT,},});After deploymentPromise me, darling, make advantage on the CloudFormation outputs.  All you need iscopy-paste,copy-paste,copy-paste, life should be always that easy.Define the following environment variables on your current session.export PROFILE_NAME="${YOUR_PROFILE_NAME}"
export JOB_ROLE_ARN="${copy-paste-thank-you}"
export APPLICATION_ID="${copy-paste-thank-you}"
export SERVERLESS_BUCKET_NAME="${copy-paste-thank-you}"
export DELTA_LAKE_SCRIPT_NAME="delta-lake-demo"Copy partial NYC-taxi data into the EMR Serverless bucket.awss3cps3://nyc-tlc/trip\data/s3://${SERVERLESS_BUCKET_NAME}/nyc-taxi/--exclude"*"--include"yellow_tripdata_2021-*.parquet"--recursive--profile${PROFILE_NAME}Create a Python script for processing Delta Laketouch${DELTA_LAKE_SCRIPT_NAME}.py
cat<< EOF > ${DELTA_LAKE_SCRIPT_NAME}.pyfrom pyspark.sql import SparkSessionimport uuidif __name__ == "__main__":"""Delta Lake with EMR Serverless, take NYC taxi as example."""spark = SparkSession \\.builder \\.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \\.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \\.enableHiveSupport() \\.appName("Delta-Lake-OSS") \\.getOrCreate()url = "s3://${SERVERLESS_BUCKET_NAME}/emr-serverless-spark/delta-lake/output/1.2.1/%s/" % str(uuid.uuid4())# creates a Delta table and outputs to target S3 bucketspark.range(5).write.format("delta").save(url)# reads a Delta table and outputs to target S3 bucketspark.read.format("delta").load(url).show()# The source for the second Delta table.base = spark.read.parquet("s3://${SERVERLESS_BUCKET_NAME}/nyc-taxi/*.parquet")# The sceond Delta table, oh ya.base.write.format("delta") \\.mode("overwrite") \\.save("s3://${SERVERLESS_BUCKET_NAME}/emr-serverless-spark/delta-lake/nyx-tlc-2021")spark.stop()EOFUpload the script and required jars into the serverless bucket# upload scriptawss3cpdelta-lake-demo.pys3://${SERVERLESS_BUCKET_NAME}/scripts/${DELTA_LAKE_SCRIPT_NAME}.py--profile${PROFILE_NAME}# download jars and upload themDELTA_VERSION="2.2.0"DELTA_LAKE_CORE="delta-core_2.13-${DELTA_VERSION}.jar"DELTA_LAKE_STORAGE="delta-storage-${DELTA_VERSION}.jar"curlhttps://repo1.maven.org/maven2/io/delta/delta-core_2.13/${DELTA_VERSION}/${DELTA_LAKE_CORE}--output${DELTA_LAKE_CORE}curlhttps://repo1.maven.org/maven2/io/delta/delta-storage/${DELTA_VERSION}/${DELTA_LAKE_STORAGE}--output${DELTA_LAKE_STORAGE}awss3mv${DELTA_LAKE_CORE}s3://${SERVERLESS_BUCKET_NAME}/jars/${${DELTA_LAKE_CORE}}--profile${PROFILE_NAME}awss3mv${DELTA_LAKE_STORAGE}s3://${SERVERLESS_BUCKET_NAME}/jars/${DELTA_LAKE_STORAGE}--profile${PROFILE_NAME}Create an EMR Serverless appRememeber, you got so much information to copy and paste from the CloudFormation outputs.awsemr-serverlessstart-job-run\--application-id${APPLICATION_ID}\--execution-role-arn${JOB_ROLE_ARN}\--name'shy-shy-first-time'\--job-driver'{"sparkSubmit": {"entryPoint": "s3://'${SERVERLESS_BUCKET_NAME}'/scripts/'${DELTA_LAKE_SCRIPT_NAME}'.py","sparkSubmitParameters": "--conf spark.executor.cores=1 --conf spark.executor.memory=4g --conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf spark.executor.instances=1 --conf spark.jars=s3://'${SERVERLESS_BUCKET_NAME}'/jars/delta-core_2.12-1.2.0.jar,s3://'${SERVERLESS_BUCKET_NAME}'/jars/delta-storage-1.2.0.jar"}}'\--configuration-overrides'{"monitoringConfiguration": {"s3MonitoringConfiguration": {"logUri": "s3://'${SERVERLESS_BUCKET_NAME}'/serverless-log/"}}}'\--profile${PROFILE_NAME}If you execute with success, you should see similar reponse as the following:{"applicationId":"00f1gvklchoqru25","jobRunId":"00f1h0ipd2maem01","arn":"arn:aws:emr-serverless:ap-northeast-1:630778274080:/applications/00f1gvklchoqru25/jobruns/00f1h0ipd2maem01"}and got a Delta Lake data unders3://${SERVERLESS_BUCKET_NAME}/emr-serverless-spark/delta-lake/nyx-tlc-2021/.Check the executing jobAccess the EMR Studio via the URL from the CloudFormation outputs. It should look very similar to the following url:https://es-pilibalapilibala.emrstudio-prod.ap-northeast-1.amazonaws.com, i.e., weird string and region won't be the same as mine.Enter into the applicationEnter into the executing jobCheck results from an EMR notebook via cluster templateCreate a workspace and an EMR cluster via the cluster template on the AWS ConsoleCheck the results delivered by the EMR serverless application via an EMR notebook.Fun factsYou can assign multiple jars as a comma-separated list to thespark.jarsasthe Spark pagesays for your EMR Serverless job. The UI will complain, you still can start the job. Don't be afraid, just click it like when you were child, facing authority fearlessly.To fully delet a stack with the construct, you need to make sure there is no more workspace within the EMR Studio. Aside from that, you also need to remove the associated identity from the Service Catalog (this is a necessary resource for the cluster template).Version inconsistency on Spark history. Possibly it can be ignored yet still made me wonder why the versions are different.So far, I still haven't figured out how to make the s3a URI work. The s3 URI is fine while the serverless app will complain that it couldn't find proper credentials provider to read the s3a URI.Future workCustom resuorce for EMR ServerlessMake the construct more flexible for usersCompare Databricks Runtime and EMR Serverless.