Package: miniwdl-aws
miniwdl AWS pluginExtendsminiwdlto run workflows onAWS BatchandEFSThis miniwdl plugin enables it to execute WDL tasks as AWS Batch jobs. It uses EFS for work-in-progress file I/O, optionally uploading final workflow outputs to S3.Before diving into this, first considerAWS HealthOmics, which includes amanaged service for WDL workflowsthat doesn't need you to provision all the infrastructure. Our companion projectminiwdl-omics-runprovides a convenient CLI for launching HealthOmics runs with local WDL source code files.There are a few ways to deploy this miniwdl-aws plugin:Amazon Genomics CLIAmazon Genomics CLIcan deploy aminiwdl-aws contextinto your AWS account with all the necessary infrastructure.Amazon SageMaker StudioOr, try theminiwdl-aws-studiorecipe to install miniwdl for interactive use withinAmazon SageMaker Studio, a web IDE with a terminal and filesystem browser. You can use the terminal to operateminiwdl runagainst AWS Batch, the filesystem browser to manage the inputs and outputs on EFS, and the Jupyter notebooks to further analyze the outputs.miniwdl-aws-submitwith custom infrastructureLastly, advanced operators can useminiwdl-aws-terraformto deploy/customize the necessary AWS infrastructure, including a VPC, EFS file system, Batch queues, and IAM roles.In this scheme, a local command-line wrapperminiwdl-aws-submitlaunches miniwdl in its own small Batch jobto orchestrate a workflow. Thisworkflow jobthen spawns WDLtask jobsas needed, without needing the submitting laptop to remain connected for the duration. The workflow jobs run on lightweightFargateresources, while task jobs run on EC2 spot instances.Submitting workflow jobsAfter deployingminiwdl-aws-terraform,pip3 install miniwdl-awslocally to make theminiwdl-aws-submitprogram available. Try the self-test:miniwdl-aws-submit --self-test --follow --workflow-queue miniwdl-workflowThen launch aviral genome assemblythat should run in 10-15 minutes:miniwdl-aws-submit \
  https://github.com/broadinstitute/viral-pipelines/raw/v2.1.28.0/pipes/WDL/workflows/assemble_refbased.wdl \
  reads_unmapped_bams=https://github.com/broadinstitute/viral-pipelines/raw/v2.1.19.0/test/input/G5012.3.testreads.bam \
  reference_fasta=https://github.com/broadinstitute/viral-pipelines/raw/v2.1.19.0/test/input/ebov-makona.fasta \
  sample_name=G5012.3 \
  --workflow-queue miniwdl-workflow \
  --s3upload s3://MY-BUCKET/assemblies \
  --verbose --followThe command line resemblesminiwdl run's with extra AWS-related arguments:--workflow-queueBatch job queue on which to schedule the workflow job; output from miniwdl-aws-terraform, defaultminiwdl-workflow. (Also set by environment variableMINIWDL__AWS__WORKFLOW_QUEUE)--followlive-streams the workflow log instead of exiting immediately upon submission. (--waitblocks on the workflow without streaming the log.)--s3upload(optional) S3 folder URI under which to upload the workflow products, including the log and output files (if successful). The bucket must be allow-listed in the miniwdl-aws-terraform deployment.Unless--s3uploadends with /, one more subfolder is added to the uploaded URI prefix, equal to miniwdl's automatic timestamp-prefixed run name. If it does end in /, then the uploads go directly into/under that folder (and a repeat invocation would be expected to overwrite them).miniwdl-aws-submitdetects other infrastructure details (task queue, EFS access point, IAM role) based on tags set on the workflow queue; seeminiwdl-aws-submit --helpfor additional options to override those defaults.Arguments not consumed byminiwdl-aws-submitarepassed throughtominiwdl runinside the workflow job; as are environment variables whose names begin withMINIWDL__, allowing override of anyminiwdl configuration option(disable wih--no-env). Seeminiwdl_aws.cfgfor various options preconfigured in the workflow job container, some of which can be adjusted to benefit specific workloads. For example, to halve the maximum rate at which miniwdl invokes the AWS Batch SubmitJob API, setMINIWDL__AWS__SUBMIT_PERIOD=2in theminiwdl-aws-submitenvironment.If the specified WDL source code is an existing local .wdl or .zip file,miniwdl-aws-submitautomatically ships it with the workflow job as the WDL to execute. Given a .wdl file, it runsminiwdl zipto detect & include any imported WDL files; while it assumes .zip files were also generated byminiwdl zip. If the source code is too large to fit in the AWS Batch request payload (~50KB), then you'll instead need to pass it by reference to a URL or EFS path.The workflow and task jobs all mount EFS at/mnt/efs. Although workflow input files are usually specified using HTTPS or S3 URIs, files already resident on EFS can be used with their/mnt/efspaths (which probably don't exist locally on the submitting machine). Unlike the WDL source code,miniwdl-aws-submitwill not attempt to ship/upload local input files.Run directories on EFSMiniwdl runs the workflow in a directory beneath/mnt/efs/miniwdl_run(override with--dir). The outputs also remain cached there for potential reuse in future runs (to avoid, submit with--no-cacheor wipe/mnt/efs/miniwdl_run/_CACHE).Given the EFS-centric I/O model, you'll need a way to browse and manage the filesystem contents remotely. The companion recipelambdash-efsis one option; miniwdl-aws-terraform outputs the infrastructure details needed to deploy it (pick any subnet). Or, set up an instance/container mounting your EFS, to access via SSH or web app (e.g.JupyterHub,Cloud Commander,VS Code server).You can also automate cleanup of EFS run directories by settingminiwdl-aws-submit --s3uploadand:--delete-after successto delete the run directory immediately after successful output upload--delete-after failureto delete the directory after failure--delete-after alwaysto delete it in either case(or set environment variableMINIWDL__AWS__DELETE_AFTER_S3_UPLOAD)Deleting a run directory after success prevents the outputs from being reused in future runs. Deleting it after failures can make debugging more difficult (although logs are retained, see below).Security note on file system isolationGoing through AWS Batch & EFS, miniwdl can't enforce the strict file system isolation between WDL task containers that it does locally. All the AWS Batch containers have read/write access to the entire EFS file system (as viewed through the access point), not only their initial working directory.This is usually benign, because WDL tasks should only read their declared inputs and write into their respective working/temporary directories. But poorly- or maliciously-written tasks could read & write files elsewhere on EFS, even changing their own input files or those of other tasks. This risks unintentional side-effects or worse security hazards from untrusted code.To mitigate this, test workflows thoroughly using the local backend, which strictly isolates task containers' file systems. If WDL tasks insist on modifying their input files in place, then--copy-input-filescan unblock them (at a cost in time, space, and IOPS). Lastly, avoid using untrusted WDL code or container images; but if they're necessary, then use a separate EFS access point and restrict the IAM and network configuration for the AWS Batch containers appropriately.EFS performance considerationsTo scale up to larger workloads, it's important to study AWS documentation on EFSperformanceandmonitoring. Like any network file system, EFS limits on throughput and IOPS can cause bottlenecks, in the worst case effectively freezing a workflow.Management tips:Monitor file system throughput limits, IOPS, and burst credits (if applicable) in the EFS area of the AWS Console.Retain the defaultElasticthroughput mode (though it may cost more than other modes)Code WDL tasks to write any purely-temporary files under/tmp, which may use local scratch space, instead of the EFS working directory.Configure miniwdl and AWS Batch to limit the number of concurrent jobs and/or the rate at which they turn over (seeminiwdl_aws.cfgfor relevant details).Spread out separate workflow runs over time or across multiple EFS file systems.FSx for Lustre and other shared filesystemsIf EFS performance remains insufficient, then you can configure your Batch compute environments to automatically mount some other shared filesystem upon instance startup. Then useminiwdl-aws-submit --no-efsto make it assume the filesystem will already be mounted at a certain location (default--mount /mnt/net) across all instances. In this case, the compute environment for workflow jobs is expected to use EC2 instead of Fargate resources (usually necessary for mounting).The miniwdl-aws-terraform repoincludes a variantsetting this up withFSx for Lustre. FSx offers higher throughput scalability, but has other downsides compared to EFS (higher upfront costs, manual capacity scaling, single-AZ deployment, fewer AWS service integrations).Logs & troubleshootingIf the terminal log isn't available (through Studio orminiwdl-submit-awsbatch --follow) to trace a workflow failure, look for miniwdl's usual log files written in the run directory on EFS or copied to S3.Each task job's log is also forwarded toCloudWatch Logsunder the/aws/batch/jobgroup and a log stream name reported in miniwdl's log. Usingminiwdl-aws-submit, the workflow job's log is also forwarded. CloudWatch Logs indexes the logs for structured search through the AWS Console & API.Misconfigured infrastructure might prevent logs from being written to EFS or CloudWatch at all. In that case, use the AWS Batch console/API to find status messages for the workflow or task jobs.Tasks can self-report their CPU & memory usage in their standard error logs, by settingMINIWDL__LOG_TASK_USAGE__PERIOD=60to report every 60 seconds (or as desired). Submit with--verbose --follow, or look in any task's CloudWatch Logs stream orstderr.txtfile, to see the "container usage" log messages.GPU jobsMiniwdl-aws recognizes thegpu: truesetting in a taskruntime{}section, and translates that to aGPU resource requirementfor AWS Batch. For the job to be scheduled, the Batch compute environment must of course make GPU instance types available.By default,gpu: truetranslates to a requirement for a single GPU. The WDL spec defines this as a boolean value, so there is no clear way to request multiple GPUs for a given task. The configurationMINIWDL__AWS__GPU_VALUEcan be set to an integerNto makealltasks withgpu: truerequireNGPUs.Alternatively, miniwdl-aws also recognizes theacceleratorTypeandacceleratorCountattributes used byAWS HealthOmics. AnyacceleratorTypestarting with "nvidia" translates to a Batch GPU requirement; the actual GPU type will depend on the instance type(s) made available by the compute environment.Multi-GPU operations may need more shared memory than Batch typically makes available in each task container. To increase the available shared memory, set e.g.MINIWDL__AWS__CONTAINER_PROPERTIES='{"linuxParameters":{"sharedMemorySize":4096}}'ContributingPull requests are welcome! For help, open an issue here or drop in on#miniwdl in the OpenWDL Slack.Code formatting and linting.To prepare your code to pass the CI checks,pip3 install --upgrade -r test/requirements.txt
pre-commit run --all-filesRunning tests.In an AWS-credentialed terminal session,MINIWDL__AWS__WORKFLOW_QUEUE=miniwdl-workflow test/run_tests.shThis builds the requisite Docker image from the current code revision and pushes it to an ECR repository (which must be prepared once withaws ecr create-repository --repository-name miniwdl-aws). To test an image from theGitHub public registryor some other version, setMINIWDL__AWS__WORKFLOW_IMAGEto the desired tag.