Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: ecs_guide.md

ECS (Elastic Container Service) tasks are a good option for executing Prefect flow runs for several reasons: Scalability: ECS scales your infrastructure in response to demand, effectively managing Prefect flow runs. ECS automatically administers container distribution across multiple instances based on demand. Flexibility: ECS lets you choose between AWS Fargate and Amazon EC2 for container operation. Fargate abstracts the underlying infrastructure, while EC2 has faster job start times and offers additional control over instance management and configuration. AWS Integration: Easily connect with other AWS services, such as AWS IAM and CloudWatch. Containerization: ECS supports Docker containers and offers managed execution. Containerization encourages reproducible deployments. Prefect enables remote flow execution via and . To learn more about these concepts please see our . For details on how workers and work pools are implemented for ECS, see the diagram below. ```mermaid %%{ init: { 'theme': 'base', 'themeVariables': { 'primaryColor': '#2D6DF6', 'primaryTextColor': '#fff', 'lineColor': '#FE5A14', 'secondaryColor': '#E04BF0', 'tertiaryColor': '#fff' } } }%% graph TB subgraph ecs_cluster[ECS cluster] subgraph ecs_service[ECS service] tdworker[Worker task definition] --> |defines| prefectworker((Prefect worker)) end prefectworker -->|kicks off| ecstask frtaskdefinition[Flow run task definition] subgraph ecs_task["ECS task execution"] style ecs_task text-align:center,display:flex flow_run((Flow run)) end frtaskdefinition -->|defines| ecs_task end subgraph prefect_cloud[Prefect Cloud] subgraph prefect_workpool[ECS work pool] workqueue[Work queue] end end subgraph github["ECR"] flow_code{{"Flow code"}} end flowcode --> |pulls| ecstask prefect_worker -->|polls| workqueue prefectworkpool -->|configures| frtask_definition ``` !!! tip "ECS tasks != Prefect tasks" An ECS task is not the same thing as a . ECS tasks are groupings of containers that run within an ECS Cluster. An ECS task's behavior is determined by its task definition. An is the blueprint for the ECS task. It describes which Docker containers to run and what you want to have happen inside these containers. ECS tasks are instances of a task definition. A Task Execution launches container(s) as defined in the task definition until they are stopped or exit on their own. This setup is ideal for ephemeral processes such as a Prefect flow run. The ECS task running the Prefect should be an , given its long-running nature and need for auto-recovery in case of failure. An ECS service automatically replaces any task that fails, which is ideal for managing a long-running process such as a Prefect worker. When a Prefect is scheduled to run it goes into the work pool specified in the flow's . are typed according to the infrastructure the flow will run on. Flow runs scheduled in an `ecs` typed work pool are executed as ECS tasks. Only Prefect ECS can poll an `ecs` typed work

When the ECS worker receives a scheduled flow run from the ECS work pool it is polling, it spins up the specified infrastructure on AWS ECS. The worker knows to build an ECS task definition for each flow run based on the configuration specified in the work pool. Once the flow run completes, the ECS containers of the cluster are spun down to a single container that continues to run the Prefect worker. This worker continues polling for work from the Prefect work pool. If you specify a task definition in the work pool, the worker will use that ARN when spinning up the ECS Task, rather than creating a task definition from the fields supplied in the work pool configuration. You can use either EC2 or Fargate as the capacity provider. Fargate simplifies initiation, but lengthens infrastructure setup time for each flow run. Using EC2 for the ECS cluster can reduce setup time. In this example, we will show how to use Fargate. <hr> !!! tip If you prefer infrastructure as code check out this to provision an ECS cluster with a worker. An AWS account with permissions to create ECS services and IAM roles. The AWS CLI installed on your local machine. You can . An to host both the worker and the flow runs it submits. This guide uses the default cluster. To create your own follow . A configured for your ECS tasks. This guide uses the default VPC. Prefect Cloud account or Prefect self-managed instance. Before setting up the worker, create a of type ECS for the worker to pull work from. If doing so from the CLI, be sure to . Create a work pool from the CLI: ```bash prefect work-pool create --type ecs my-ecs-pool ``` Or from the Prefect UI: !!! Because this guide uses Fargate as the capacity provider and the default VPC and ECS cluster, no further configuration is needed. Next, set up a Prefect ECS worker that will discover and pull work from this work pool. First start by creating the required in order for your worker and flows to run. The sample flow in this guide doesn't interact with many other AWS services, so you will only be creating one role, `taskExecutionRole`. To create an IAM role for the ECS task using the AWS CLI, follow these steps: The trust policy will specify that the ECS service containing the Prefect worker will be able to assume the role required for calling other AWS services. Save this policy to a file, such as `ecs-trust-policy.json`: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` Use the `aws iam create-role` command to create the roles that you will be

For this guide, the `ecsTaskExecutionRole` will be used by the worker to start ECS tasks, and will also be the role assigned to the ECS tasks running your Prefect flows. ```bash aws iam create-role \ --role-name ecsTaskExecutionRole \ --assume-role-policy-document file://ecs-trust-policy.json ``` !!! tip Depending on the requirements of your flows, it is advised to create a . This role will contain the permissions required by the ECS tasks in which your flows will run. For example, if your workflow loads data into an S3 bucket, you would need a role with additional permissions to access S3. For this guide the ECS worker will require permissions to pull images from ECR and publish logs to CloudWatch. Amazon has a managed policy named `AmazonECSTaskExecutionRolePolicy` that grants the permissions necessary for starting ECS tasks. for other common execution role permissions. Attach this policy to your task execution role: ```bash aws iam attach-role-policy \ --role-name ecsTaskExecutionRole \ --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy ``` Remember to replace the `--role-name` and `--policy-arn` with the actual role name and policy Amazon Resource Name (ARN) you want to use. Next, create an ECS task definition that specifies the Docker image for the Prefect worker, the resources it requires, and the command it should run. In this example, the command to start the worker is `prefect worker start --pool my-ecs-pool`. Create a JSON file with the following contents: ```json { "family": "prefect-worker-task", "networkMode": "awsvpc", "requiresCompatibilities": [ "FARGATE" ], "cpu": "512", "memory": "1024", "executionRoleArn": "<ecs-task-role-arn>", "taskRoleArn": "<ecs-task-role-arn>", "containerDefinitions": [ { "name": "prefect-worker", "image": "prefecthq/prefect:3-latest", "cpu": 512, "memory": 1024, "essential": true, "command": [ "/bin/sh", "-c", "pip install prefect-aws && prefect worker start --pool my-ecs-pool --type ecs" ], "environment": [ { "name": "PREFECTAPIURL", "value": "prefect-api-url>" }, { "name": "PREFECTAPIKEY", "value": "<prefect-api-key>" } ] } ] } ``` Use `prefect config view` to view the `PREFECTAPIURL` for your current Prefect profile. Use this to replace `<prefect-api-url>`. For the `PREFECTAPIKEY`, if you are on a paid plan you can create a for the worker. If your are on a free plan, you can pass a users API key. Replace both instances of `<ecs-task-role-arn>` with the ARN of the IAM role you created in Step 2. You can grab this by running: ``` aws iam get-role --role-name taskExecutionRole --query 'Role.[RoleName, Arn]' --output text ``` Notice that the CPU and Memory allocations are relatively small. The worker's main responsibility is to submit work through API calls to AWS, not to execute your Prefect flow code. !!! tip To avoid hardcoding your API key into the task definition JSON see . Before creating a service, you first need to register a task definition. You can do that using the `register-task-definition` command in the AWS CLI. Here is an example: ```bash aws ecs register-task-definition --cli-input-json file://task-definition.json ``` Replace `task-definition.json` with the name of your JSON

Finally, create a service that will manage your Prefect worker: Open a terminal window and run the following command to create an ECS Fargate service: ```bash aws ecs create-service \ --service-name prefect-worker-service \ --cluster <ecs-cluster> \ --task-definition <task-definition-arn> \ --launch-type FARGATE \ --desired-count 1 \ --network-configuration "awsvpcConfiguration={subnets=[<subnet-ids>],securityGroups=[<security-group-ids>],assignPublicIp='ENABLED'}" ``` Replace `<ecs-cluster>` with the name of your ECS cluster. Replace `<task-definition-arn>` with the ARN of the task definition you just registered. Replace `<subnet-ids>` with a comma-separated list of your VPC subnet IDs. Ensure that these subnets are aligned with the vpc specified on the work pool in step 1. You can view subnet ids with the following command: `aws ec2 describe-subnets --filter Name=<vpc-id>` Replace `<security-group-ids>` with a comma-separated list of your VPC security group IDs. !!! tip "Sanity check" The work pool page in the Prefect UI allows you to check the health of your workers - make sure your new worker is live! Note that it can take a few minutes for an ECS service to come online. If your worker does not come online and you are using the command from this guide, you may not be using the default VPC. For connectivity issues, check your VPC's configuration and refer to the . This guide uses ECR to store a Docker image containing your flow code. To do this, we will write a flow, then deploy it using build and push steps that copy flow code into a Docker image and push that image to an ECR repository. `my_flow.py` ```python from prefect import flow, getrunlogger @flow def my_flow(): logger = getrunlogger() logger.info("Hello from ECS!!") if name == "main": my_flow() ``` Use the following AWS CLI command to create an ECR repository. The name you choose for your repository will be reused in the next step when defining your Prefect deployment. ```bash aws ecr create-repository \ --repository-name <my-ecr-repo> \ --region <region> ``` To have Prefect build your image when deploying your flow create a `prefect.yaml` file with the following specification: ```yaml name: ecs-worker-guide prefect-version: 2.14.20 build: prefectdocker.deployments.steps.builddocker_image: id: build_image requires: prefect-docker>=0.3.1 image_name: <my-ecr-repo> tag: latest dockerfile: auto push: prefectdocker.deployments.steps.pushdocker_image: requires: prefect-docker>=0.3.1 imagename: '{{ buildimage.image_name }}' tag: '{{ build_image.tag }}' deployments: name: myecsdeployment version: tags: [] description: entrypoint: flow.py:my_flow parameters: {} work_pool: name: ecs-dev-pool workqueuename: job_variables: image: '{{ build_image.image }}' schedules: [] pull: prefect.deployments.steps.setworkingdirectory: directory: /opt/prefect/ecs-worker-guide ``` ```bash prefect deploy myflow.py:myecs_deployment ``` Now that you are confident your ECS worker is healthy, you can experiment with different work pool configurations. Do your flow runs require higher `CPU`? Would an EC2 `Launch Type` speed up your flow run execution? These infrastructure configuration values can be set on your ECS work pool or they can be overridden on the deployment level through if desired.