Package: emr-cli
EMR CLISo we're all working on data pipelines every day, but wouldn't be nice to just hit a button and have our code automatically deployed to staging or test accounts? I thought so, too, thats why I created the EMR CLI (emr) that can help you package and deploy your EMR jobs so you don't have to.The EMR CLI supports a wide variety of configuration options to adapt toyourdata pipeline, not the other way around.Packaging - Ensure a consistent approach to packaging your production Spark jobs.Deployment - Easily deploy your Spark jobs across multiple EMR environments or deployment frameworks like EC2, EKS, and Serverless.CI/CD - Easily test each iteration of your code without resorting to messy shell scripts. :)The initial use cases are:Consistent packaging for PySpark projects.Use in CI/CD pipelines for packaging, deployment of artifacts, and integration testing.Warning: This tool is still under active development, so commands may change until a stable 1.0 release is made.Quick StartYou can use the EMR CLI to take a project from nothing to running in EMR Serverless is 2 steps.First, let's install theemrcommand.python3-mpipinstall-Uemr-cliNoteThis tutorial assumes you have alreadysetup EMR Serverlessand have an EMR Serverless application, job role, and S3 bucket you can use. If not, you can use theemr bootstrapcommand.Create a sample projectemrinitscratch📔 Tip: Use--project-type poetryto create aPoetryproject!You should now have a sample PySpark project in your scratch directory.scratch
├── Dockerfile
├── entrypoint.py
├── jobs
│   └── extreme_weather.py
└── pyproject.toml

1 directory, 4 filesNow deploy and run on an EMR Serverless application!emrrun\--entry-pointentrypoint.py\--application-id${APPLICATION_ID}\--job-role${JOB_ROLE_ARN}\--s3-code-uris3://${S3_BUCKET}/tmp/emr-cli-demo/\--s3-logs-uris3://${S3_BUCKET}/logs/emr-cli-demo/\--build\--show-stdoutThis command performs the following actions:Packages your project dependencies into a Python virtual environmentUploads the Spark entrypoint and packaged dependencies to S3Starts an EMR Serverless jobWaits for the job to run to completion and shows thestdoutof the Spark driver when finished!And you're done. Feel free to modify the project to experiment with different things. You can simply re-run the command above to re-package and re-deploy your job.EMR CLI Sub-commandsThe EMR CLI has several subcommands that you can see by runningemr --helpCommands:
  bootstrap  Bootstrap an EMR Serverless environment.
  deploy     Copy a local project to S3.
  init       Initialize a local PySpark project.
  package    Package a project and dependencies into dist/
  run        Run a project on EMR, optionally build and deploy
  statusbootstrapemr bootstrapallows you to create a sample EMR Serverless or EMR on EC2 environment for testing. It assumes you have admin access and creates various resources for you using AWS APIs.EMR ServerlessTo create a bootstrap EMR Serverless environment, using the following command:emrbootstrap\--targetemr-serverless\--code-bucket<your_unique_new_bucket_name>\--job-role-name<your_unique_emr_serverless_job_role_name>When you do this, the CLI creates a new EMR CLI config file at.emr/config.yamlthat will set default locations for youremr runcommand.initTheinitcommand creates a newpyproject.tomlorpoetryproject for you with a sample PySpark application.initis required to create those project types as it also initializes aDockerfileused to package your dependencies. Single-file PySpark jobs and simple Python modules do not require theinitcommand to be used.packageThepackagecommand bundles your PySpark code and dependencies in preparation for deployment. Often you'll either usepackageanddeployto deploy new artifacts to S3, or you'll use the--buildflag in theemr runcommand to handle both of those tasks for you.The EMR CLI automatically detects what type of project you have and builds the necessary dependency packages.deployThedeploycommand copies the project dependencies from thedist/folder to your specified S3 location.runTheruncommand is intended to help package, deploy, and run your PySpark code across EMR on EC2, EMR on EKS, or EMR Serverless.You must provide one of--cluster-id,--virtual-cluster-id, or--application-idto specify which environment to run your code on.emr run --helpshows all the available options:Usage: emr run [OPTIONS]

  Run a project on EMR, optionally build and deploy

Options:
  --application-id TEXT         EMR Serverless Application ID
  --cluster-id TEXT             EMR on EC2 Cluster ID
  --virtual-cluster-id TEXT     EMR on EKS Virtual Cluster ID
  --entry-point FILE            Python or Jar file for the main entrypoint
  --job-role TEXT               IAM Role ARN to use for the job execution
  --wait                        Wait for job to finish
  --s3-code-uri TEXT            Where to copy/run code artifacts to/from
  --s3-logs-uri TEXT            Where to send EMR Serverless logs to
  --job-name TEXT               The name of the job
  --job-args TEXT               Comma-delimited string of arguments to be
                                passed to Spark job

  --spark-submit-opts TEXT      String of spark-submit options
  --build                       Package and deploy job artifacts
  --show-stdout                 Show the stdout of the job after it's finished
  --save-config                 Update the config file with the provided
                                options

  --emr-eks-release-label TEXT  EMR on EKS release label (emr-6.15.0) -
                                defaults to latest releaseSupport PySpark configurationsSingle-file project - Projects that have a single.pyentrypoint file.Multi-file project - A more typical PySpark project, but without dependencies, that has multiple Python files or modules.Python module - A project with dependencies defined in apyproject.tomlfile.Poetry project - A project usingPoetryfor dependency management.Sample CommandsCreate a new PySpark project (other frameworks TBD)emrinitproject-dirPackage your project into a virtual environment archiveemrpackage--entry-pointmain.pyThe EMR CLI auto-detects the project type and will change the packaging method appropriately.If you have additional.pyfiles, those will be included in the archive.Deploy an existing package artifact to S3.emrdeploy--entry-pointmain.py--s3-code-uris3://<BUCKET>/code/Deploy a PySpark package to S3 and trigger an EMR Serverless jobemrrun--entry-pointmain.py\--s3-code-uris3://<BUCKET>/code/\--application-id<EMR_SERVERLESS_APP>\--job-role<JOB_ROLE_ARN>Build, deploy, and run an EMR Serverless job and wait for it to finish.emrrun--entry-pointmain.py\--s3-code-uris3://<BUCKET>/code/\--application-id<EMR_SERVERLESS_APP>\--job-role<JOB_ROLE_ARN>\--build\--waitRe-run an already deployed job and show thestdoutof the driver.emrrun--entry-pointmain.py\--s3-code-uris3://<BUCKET>/code/\--s3-logs-uris3://<BUCKET>/logs/\--application-id<EMR_SERVERLESS_APP>\--job-role<JOB_ROLE_ARN>\--show-stdoutNote: If the job fails, the command will exit with an error code.Re-run your jobs with 7 characters.If you provide the--save-configcommand toemr run, it will save a configuration file for you in.emr/config.yamland next time you can useemr runwith no parameters to re-run your job.emrrun--entry-pointmain.py\...\--save-config[emr-cli]:Configfilesavedto.emr/config.yaml.Use`emrrun`tore-useyourconfiguration.❯emrrun[emr-cli]:Usingconfigfile:.emr/config.yaml🥳Run the same job against an EMR on EC2 cluster```bash
emrrun--entry-pointmain.py\--s3-code-uris3://<BUCKET>/code/\--s3-logs-uris3://<BUCKET>/logs/\--cluster-id<EMR_EC2_CLUSTER_ID>--show-stdoutOr an EMR on EKS virtual cluster.emrrun--entry-pointmain.py\--s3-code-uris3://<BUCKET>/code/\--s3-logs-uris3://<BUCKET>/logs/\--virtual-cluster-id<EMR_EC2_CLUSTER_ID>\--job-role<EMR_EKS_JOB_ROLE_ARN>\--show-stdoutSecuritySeeCONTRIBUTINGfor more information.LicenseThis project is licensed under the Apache-2.0 License.