Package: aws-emr-launch
EMR LaunchAnAWS Professional Serviceopen source initiative |aws-proserve-opensource@amazon.comThe intent of the EMR Launch library is to simplify the development experience for Builders defining, deploying, managing, and using EMR Clusters by:defining reusable Security, Resource, and Launch Configurations enabling developers toDefine Once and Reuseseparating the definition of Cluster Security Configurations and Cluster Resource Configurations into reusable and shareable Constructsproviding a suite of Tools to simplify the construction of Orchestration Pipelines using Step Functions and EMR ClustersConcepts (and Constructs)This library utilizes the AWS CDK for deployment and management of resources. It is recommended that users familiarize themselves with the CDK's basic concepts and usage.EMR ProfileAn EMR Profile (emr_profile) is a reusable definition of the security profile used by an EMR Cluster. This includes:Service Role: an IAM Role used by the EMR Service to manage the ClusterInstance Role: an IAM Role used by the EC2 Instances in an EMR ClusterAutoScaling Role: an IAM Role used to autoscale and resize an EMR ClusterService Group: a Security Group granting the EMR Service basic access to EC2 Instances in Cluster. This is required to deploy Instances into a Private Subnet.Master Group: the Security Group assigned to the EMR Master InstanceWorkers Group: the Security Group assigned to the EMR Worker Instances (Core and Task nodes)Security Configuration: the Security Configuration used by the ClusterKerberos Attributes: the attributes required to enable Kerberos authenticationEachemr_profilerequires a uniqueprofile_name. This name and thenamespaceuniquely identify a profile. Thenamespaceis a logical grouping of profiles and has a default value of "default".Deploying anemr_profilecreates these resources and stores the profile definition and metadata in the Parameter Store. The Profile can either be used immediately in the Stack when it is defined, or reused in other Stacks by loading the Profile definition byprofile_nameandnamespace.Cluster ConfigurationA Cluster Configuration (cluster_configuration) is a reusable definition of the physical resources in an EMR Cluster. This incudes:EMR Release Label: the EMR release version (e.g.emr-5.28.0)Applications: the Applications to install on Cluster (e.g. Hadoop, Hive, SparK)Bootstrap Actions: the Bootstrap Actions to execute on each node after Applications have been installedConfigurations: configuration parameters to set for the various Applications installedStep Concurrency Level: the number of concurrent Steps the Cluster is configured to runInstances: the configuration of the Master, Core, and Task nodes in the Cluster (e.g. Master Instance Type, Core Instance Type, Core Instance Count, etc)Like theemr_profile, eachcluster_configurationrequires a uniqueconfiguration_name. This name and thenamespaceuniquely identify a configuration.Deploying acluster_configurationstores the configuration definition and metadata in the Parameter Store. The Configuration can either be used immediately in the Stack when it is defined, or reused in other Stacks by loading the Configuration definition byconfiguration_nameandnamespace.EMR Launch FunctionAn EMR Launch Function (emr_launch_function) is an AWS Step Functions State Machine that launches an EMR Cluster. The Launch Function is defined with anemr_profile,cluster_configuration,cluster_name, andtags. When the function is executed it creates an EMR Cluster with the given name, tags, security profile, and physical resources then synchronously monitors the cluster for successful start.To be clear, deploying anemr_launch_functiondoes notcreate an EMR Cluster, it only creates the State Machine. The cluster is created when the State Machine is executed.Theemr_launch_functionis a mechanism for easily combining the reusableemr_profileandcluster_configuration.Like theemr_profileandcluster_configuration, eachemr_launch_functionrequires a uniquelaunch_function_name. This name and thenamespaceuniquely identify the launch function.Chains and TasksChains and Tasks are preconfigured components that simplify the use of AWS Step Function State Machines as orchestrators of data processing pipelines. These components allow the developer to easily build complex, serverless pipelines using EMR Clusters (both Transient and Persistent), Lambdas, and nested State Machines.SecurityCare is taken to ensure thatemr_launch_functionsandemr_profilescan't be used to create clusters with elevated or unintended privileges.IAM policies can be used to restrict the Users and Roles that can create EMR Clusters by grantingstates:StartExecutionto specific State Machine ARNs.By storing the metadata and configuration ofemr_profiles,cluster_configurations, andemr_launch_functionsin the Systems Manager Parameter Store, IAM Policies can be used to grant or restrict Read/Write access to theseAccess can be managed forALLmetadata and configurations, specificnamespaces, or individual ARNsEachemr_launch_functionuses a specific AWS Lambda function to load and combine its specificemr_profileandcluster_configuration. The IAM Policy associated with this Lambda allows it to read only these specific ARNs from the Parameter Store.Eachemr_launch_functionis grantediam:PassRoleto the specific EMR Roles defined in theemr_profileassigned to the launch function. Attempting to change the Roles used by directly modifying the metadata of theemr_profilein the Parameter Store will result in a cluster launch failure.UsageThis library acts as a plugin to theAWS CDKproviding additional L2 Constructs.
To avoid circular references with CDK dependencies this package will not install CDK and Boto3. These should be
installed manually from one of therequirements.txtfiles (depending on the version ofaws-emr-launch).It is recommended that a Python3venvbe used for all CDK builds and deployments.To get up and running quickly:PrerequisitesThe AWS CDK v2.x utilizes containers to automate some tasks. EMR Launch uses and deploys a CDKPythonLayerVersion, this Construct uses a container to create the bundle for the Lambda Layer. As such, adockerruntime is required to deploy.DeploymentInstall theCDK CLInpminstall-gaws-cdkUse your mechanism of choice to create and activate a Python3venv:python3-mvenv.envsource.env/bin/activateInstall the CDK and Boto3 minimum requirements:pipinstall-rrequirements-2.x.txtInstallaws-emr-launchpackage:pipinstallaws-emr-launchDevelopmentFollow Steps 1 - 3 above to configure an environment and install requirementsAfter activating yourvenv:Install development requirements:pipinstall-rrequirements-dev.txtInstall the library locally:pipinstall-e.Managing Layer PackagesUpdate theaws_emr_launch/lambda_sources/layers/emr_config_utils/requirements.txtadding/updating/removing package(s)TestingTo run the test suite (from within thevenv):pytestAfter running testsView test coverage reports by openinghtmlcov/index.htmlin your web browser.To write a teststart a file named test_[the module you want to test].pyimport the module you want to test at the top of the filewrite test case functions that match eithertest_*or*_testFor more information refer topytest docsContributingSeeCONTRIBUTINGfor more information.LicenseThis project is licensed under the terms of the Apache 2.0 license. SeeLICENSE.
Included AWS Lambda functions are licensed under the MIT-0 license. SeeLICENSE-LAMBDA.