Repository: PrefectHQ/prefect

Issue:
`ECSRun` fails with boto `InvalidParameterException` needs awsvpc network mode
## Opened from the [Prefect Public Slack Community](https://prefect-slackin.herokuapp.com/)

**leeca.jinlee**: Hello all, trying out the latest Prefect version `0.14.12`

Running into this error when attempting to run a flow using ECS agent and ECSRun:

`botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.`

I have a working config for prefect agent that executes the flow without errors. However, this involves creating a `task-definitions.yaml`:

```
 prefect agent ecs start -t token \
    -n aws-ecs-agent \
    -l label \
    --task-definition /path/to/task-definition.yaml \
    --cluster cluster_arn
```

`task-definitions.yaml`

```
networkMode: awsvpc
cpu: 1024
memory: 2048
taskRoleArn: task_role_arn
executionRoleArn: execution_role_arn
```

The flow runs without errors, so the error is not due to IAM permissions.

However, when running the ECS Agent using the `--task-role-arn` and `--execution-role-arn` CLI args, I run into the above-mentioned error. I have also tried running Prefect agent using `--launch-type FARGATE` , which I believe is the default and does not need to be specified, but this does not work too.

```
prefect agent ecs start -t token \
    -n aws-ecs-agent \
    -l ecs \
    --task-role-arn task_role_arn \
    --execution-role-arn execution_role_arn \
    --cluster cluster_arn
```

I have also tried to pass in `task_role_arn` and `execution_role_arn` into the ECSRun() function within my flow, and ran into the same error.

Is there any way to run ECS Agent using CLI args without using the task-definition file?

**leeca.jinlee**: In the AWS docs for Task Definition, under <https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-networkmode|Network Mode>, it says:

&gt; If you are using the Fargate launch type, the `awsvpc` network mode is required.
However, it does not seem that there is a cli arg to pass to `prefect agent ecs start`.

I have also tried passing in a `networkConfiguration`  dict to `run_task_kwargs` arg in ECSRun():


```
"networkConfiguration": {
    "awsvpcConfiguration": {
        'assignPublicIp': 'ENABLED', 
        'subnets': ['subnet-1', 'subnet-2', 'subnet-3'], 
        'securityGroups': []
    }
}
```

Still running into the same error. Seems that the agent needs to know about `awsvpc` as the network mode, but there doesn't seem to be a way to tell it without using a task definitions file

**michael054**: Hey <@U01AYG8QZ4Y>, thanks for the thorough explanation.  I'm going to open an issue for this in the Prefect Core repo as this looks like it may need a PR to address. <@ULVA73B9P> open "`ECSRun` fails with boto `InvalidParameterException` needs awsvpc network mode"

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1615453867147100?thread_ts=1615453867.147100&cid=CL09KU1K7).




Discussion:
Additional info reported at https://prefect-community.slack.com/archives/CL09KU1K7/p1615830767346000

latest task definition
```
  "compatibilities": [
    "EC2"
  ],

```
with 0.14.6
```
"compatibilities": [
    "EC2",
    "FARGATE"
  ],
```
From what I can tell the task definition submitted to ECS for 0.14.12 differs significantly from the task definition submitted for versions < 0.14.12.

Given this simple flow: 
```
import prefect
from prefect.storage import S3
from prefect.run_configs import ECSRun
from prefect import task, Flow

RUN_CONFIG = ECSRun(
    image= '{redacted}.dkr.ecr.us-east-1.amazonaws.com/prefect-aws:latest',
    labels=["s3-flow-storage"],
    memory="512",
    cpu="256",
)
STORAGE = S3(bucket="prefect-rai-dev", stored_as_script=True)

@task
def say_hello():
    logger = prefect.context.get("logger")
    logger.info(f"Hello from prefect!")

with Flow("say_hello_flow", storage=STORAGE, run_config=RUN_CONFIG) as flow:
    say_hello()
```

Here is the task definition for 0.14.11 submitted by the Agent via boto3: 
```
{
  "ipcMode": null,
  "executionRoleArn": "arn:aws:iam::{redacted}:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": null,
      "entryPoint": null,
      "portMappings": [],
      "command": [
        "/bin/sh",
        "-c",
        "prefect execute flow-run"
      ],
      "linuxParameters": null,
      "cpu": 0,
      "environment": [
        {
          "name": "PREFECT__CLOUD__USE_LOCAL_SECRETS",
          "value": "false"
        },
        {
          "name": "PREFECT__CONTEXT__IMAGE",
          "value": "{redacted}.dkr.ecr.us-east-1.amazonaws.com/prefect-aws:latest"
        },
        {
          "name": "PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS",
          "value": "prefect.engine.cloud.CloudFlowRunner"
        },
        {
          "name": "PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS",
          "value": "prefect.engine.cloud.CloudTaskRunner"
        }
      ],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "{redacted}.dkr.ecr.us-east-1.amazonaws.com/prefect-aws:latest",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "flow"
    }
  ],
  "placementConstraints": [],
  "memory": "512",
  "taskRoleArn": "arn:aws:iam::{redacted}:role/Prefect_Container_Role",
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "arn:aws:ecs:us-east-1:{redacted}:task-definition/prefect-say-hello-flow:10",
  "family": "prefect-say-hello-flow",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-ecr-pull"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [],
  "networkMode": "awsvpc",
  "cpu": "256",
  "revision": 10,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}
```

Here is the task definition for 0.14.12 submitted by the Agent via boto3: 
```
{
  "ipcMode": null,
  "executionRoleArn": null,
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": null,
      "entryPoint": null,
      "portMappings": [],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [
        {
          "name": "PREFECT__CONTEXT__IMAGE",
          "value": "{redacted}.dkr.ecr.us-east-1.amazonaws.com/prefect-aws:latest"
        }
      ],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "{redacted}.dkr.ecr.us-east-1.amazonaws.com/prefect-aws:latest",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "flow"
    }
  ],
  "placementConstraints": [],
  "memory": "2048",
  "taskRoleArn": null,
  "compatibilities": [
    "EC2"
  ],
  "taskDefinitionArn": "arn:aws:ecs:us-east-1:{redacted}:task-definition/prefect-say-hello-flow:11",
  "family": "prefect-say-hello-flow",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "revision": 11,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}
```

The most significant changes in my view are the missing execution role and task role arns and compatibilities section. From what I can tell, if the executionRoleArn is not populated the compatibilities section automatically complied by ECS will only have "EC2" as an option. Submitting an executionRoleArn, even a phony one will get ECS to add "FARAGE" to the compatibilities section.

I can work-around this issue by manually adding a task definition to the run config as shown below: 

```
import prefect
from prefect.storage import S3
from prefect.run_configs import ECSRun
from prefect import task, Flow

import yaml

definition = yaml.safe_load(
    """
    networkMode: awsvpc
    cpu: 1024
    memory: 2048
    containerDefinitions:
        - name: flow
    executionRoleArn: aws:iam::{redacted}:role/Prefect_Task_Execution_Role
    """
)

RUN_CONFIG = ECSRun(
    image= '{redacted}.dkr.ecr.us-east-1.amazonaws.com/prefect-aws:latest',
    task_definition=definition,
    labels=["s3-flow-storage"],
    memory="512",
    cpu="256",
)
STORAGE = S3(bucket="prefect-rai-dev", stored_as_script=True)

@task
def say_hello():
    logger = prefect.context.get("logger")
    logger.info(f"Hello from prefect!")

with Flow("say_hello_flow", storage=STORAGE, run_config=RUN_CONFIG) as flow:
    say_hello()
```

Please note that the task execution role specified above DOES NOT EXIST, yet works.
Please note that the Agent is currently specifying both the executionRoleArn and taskRoleArn.
Please note that adding the executionRoleArn and taskRoleArn directly to the Run Config doesn't solve the problem.

In addition, there was some question about how the ECS/Fargate cluster was created. Originally we had created the cluster manually via the CLI. We did update our cluster definition to include the FARGATE and FARGATE_SPOT capacity providers and we added a default capacity provider strategy. None of those changes made any difference. 

We also went back and created a new cluster using the "Getting Started" UI flow from the AWS console. This also didn't make any difference.

Lastly, on a lark, we changed our task execution role to be exactly "ecsTaskExecutionRole" thinking there might be some magic in the naming and that also didn't make any difference.

To be complete with information here is our Agent Dockerfile: 

```
FROM python:3.9-slim-buster

ENV PREFECT_VERSION=0.14.12

RUN apt-get update && apt-get install -y gcc

RUN pip install prefect[aws]==${PREFECT_VERSION}

COPY agent.py /agent.py
COPY --from=arpaulnet/s6-overlay-stage:2.0 / /

ENTRYPOINT ["/init"]

CMD ["python", "agent.py"]
```

Here is the agent.py: 
```
import prefect
from prefect.agent.ecs import ECSAgent

TASK_ROLE_ARN = prefect.config.task.role.arn
EXECUTION_ROLE_ARN = prefect.config.execution.role.arn

logger = prefect.context.get("logger")
logger.info(f"task_role_arn={TASK_ROLE_ARN}, execution_role_arn={EXECUTION_ROLE_ARN}")

ECSAgent(
    name="RAI ECS/Fargate Agent",
    cluster="RAI",
    task_role_arn=TASK_ROLE_ARN,
    execution_role_arn=EXECUTION_ROLE_ARN,
    env_vars={"PREFECT__CONSUL__DNS": "consul.dev.drillinginfo.com"},
).start()
```

Here is the ECS cluster description: 
```
{
    "clusters": [
        {
            "clusterArn": "arn:aws:ecs:us-east-1:{redacted}:cluster/RAI",
            "clusterName": "RAI",
            "status": "ACTIVE",
            "registeredContainerInstancesCount": 0,
            "runningTasksCount": 0,
            "pendingTasksCount": 1,
            "activeServicesCount": 0,
            "statistics": [],
            "tags": [],
            "settings": [
                {
                    "name": "containerInsights",
                    "value": "enabled"
                }
            ],
            "capacityProviders": [
                "FARGATE_SPOT",
                "FARGATE"
            ],
            "defaultCapacityProviderStrategy": []
        }
    ],
    "failures": []
}
```

Resolving patch:
diff --git a/src/prefect/agent/ecs/agent.py b/src/prefect/agent/ecs/agent.py
--- a/src/prefect/agent/ecs/agent.py
+++ b/src/prefect/agent/ecs/agent.py
@@ -405,6 +405,20 @@ def generate_task_definition(
         if "memory" in taskdef:
             taskdef["memory"] = str(taskdef["memory"])
 
+        # Set executionRoleArn if configured
+        # Note that we set this in both the task definition and the
+        # run_task_kwargs since ECS requires this in the definition for FARGATE
+        # tasks, but we also want to still configure it for users that provide
+        # their own `task_definition_arn`.
+        if run_config.execution_role_arn:
+            taskdef["executionRoleArn"] = run_config.execution_role_arn
+        elif self.execution_role_arn:
+            taskdef["executionRoleArn"] = self.execution_role_arn
+
+        # Set requiresCompatibilities if not already set
+        if "requiresCompatibilities" not in taskdef:
+            taskdef["requiresCompatibilities"] = [self.launch_type]
+
         return taskdef
 
     def get_run_task_kwargs(
@@ -421,8 +435,7 @@ def get_run_task_kwargs(
         """
         # Set agent defaults
         out = deepcopy(self.run_task_kwargs)
-        if self.launch_type:
-            out["launchType"] = self.launch_type
+        out["launchType"] = self.launch_type
         if self.cluster:
             out["cluster"] = self.cluster
 


Test patch:
diff --git a/tests/agent/test_ecs_agent.py b/tests/agent/test_ecs_agent.py
--- a/tests/agent/test_ecs_agent.py
+++ b/tests/agent/test_ecs_agent.py
@@ -326,6 +326,28 @@ def test_generate_task_definition_family_and_tags(self):
             {"key": "prefect:flow-version", "value": "1"},
         ]
 
+    @pytest.mark.parametrize("launch_type", [None, "FARGATE", "EC2"])
+    def test_generate_task_definition_requires_compatibilities(self, launch_type):
+        taskdef = self.generate_task_definition(ECSRun(), launch_type=launch_type)
+        assert taskdef["requiresCompatibilities"] == [launch_type or "FARGATE"]
+
+    @pytest.mark.parametrize(
+        "on_run_config, on_agent, expected",
+        [
+            (None, None, None),
+            ("execution-role-1", None, "execution-role-1"),
+            (None, "execution-role-2", "execution-role-2"),
+            ("execution-role-1", "execution-role-2", "execution-role-1"),
+        ],
+    )
+    def test_get_task_run_kwargs_execution_role_arn(
+        self, on_run_config, on_agent, expected
+    ):
+        taskdef = self.generate_task_definition(
+            ECSRun(execution_role_arn=on_run_config), execution_role_arn=on_agent
+        )
+        assert taskdef.get("executionRoleArn") == expected
+
     @pytest.mark.parametrize(
         "run_config, storage, expected",
         [