Repository: PrefectHQ/prefect

Issue:
Pass private registry creds to Fargate agent (non-programmatic approach)
## Current behavior

I'm trying to execute flows, configured without explicit specification of environment (e.g. `FargateEnvironment`) through the generic agent as specified below:

```yaml
  my-agent:
    image: prefecthq/prefect:${PREFECT_VERSION}
    environment:
      - PREFECT__BACKEND=server
      - PREFECT__SERVER__HOST=http://x.x.x.x
      - PREFECT__LOGGING__LEVEL=DEBUG
      - PREFECT__FLOWS__CHECKPOINTING=true
      - AWS_ACCESS_KEY_ID=...
      - AWS_SECRET_ACCESS_KEY=...
      - REGION_NAME=eu-central-1
      - containerDefinitions_logConfiguration={"logDriver":"awslogs","options":{"awslogs-group":"npn/prefect","awslogs-region":"eu-central-1","awslogs-stream-prefix":"flow-runs"}}
    restart: always
    command: >
      prefect agent start fargate
        --name my-test-agent
        --show-flow-logs
        -l fargate
        -v
        cpu=256
        memory=512
        executionRoleArn=arn:aws:iam::********:role/ecsTaskExecutionRole
        cluster=npn-ecs1
        networkConfiguration="{'awsvpcConfiguration': {'assignPublicIp': 'DISABLED', 'subnets': ['subnet-*******'], 'securityGroups': ['sg-*******']}}"
```

Everything works as expected excepting pulling image from our private docker registry. The documentation for Fargate and Boto3 suggests to put access token into AWS Secret Manager (done) and pass it in the following structure:

```python
containerDefinitions = {
   "repositoryCredentials": {
        "credentialsParameter":"arn:aws:secretsmanager:eu-central-1:*****:secret:<my secret>"}
}
```

But I couldn't find a way to pass such a parameter via environment variables or **kwargs to agent as CLI parameters.

## Proposed behavior

Would it be possible to add support for credentials in the same way how it works for `containerDefinitions_logConfiguration` or `networkConfiguration`?

E.g.:
```yaml
    environment:
      - PREFECT__BACKEND=server
      - PREFECT__SERVER__HOST=http://x.x.x.x
      - PREFECT__LOGGING__LEVEL=DEBUG
      - PREFECT__FLOWS__CHECKPOINTING=true
      - AWS_ACCESS_KEY_ID=...
      - AWS_SECRET_ACCESS_KEY=...
      - REGION_NAME=eu-central-1
      - containerDefinitions_logConfiguration={"logDriver":"awslogs","options":{"awslogs-group":"npn/prefect","awslogs-region":"eu-central-1","awslogs-stream-prefix":"flow-runs"}}
      - containerDefinitions_repositoryCredentials={"credentialsParameter":"arn:aws:secretsmanager:eu-central-1:*****:secret:<my secret>"}
```




Discussion:
@oleksandr checkout the `fargate_repo_creds` branch and let me know if that works for you https://github.com/PrefectHQ/prefect/compare/fargate_repo_creds
@joshmeek thanks! checking it out...
@joshmeek OK, I see in the logs:
```
fargate-agent_1        | [2020-06-19 14:33:26,723] DEBUG - dev-layout-fargate-agent | Container definition: logConfiguration from environment variable
fargate-agent_1        | [2020-06-19 14:33:26,723] DEBUG - dev-layout-fargate-agent | Container definition: repositoryCredentials from environment variable
```

which looks good... I proceed to executing the flow and see the following in the logs:

```
fargate-agent_1        | [2020-06-19 14:35:33,645] DEBUG - dev-layout-fargate-agent | Run created for task arn:aws:ecs:eu-central-1:**********:task/0242fe4f-8227-4705-8724-e85e81bd6668
fargate-agent_1        | [2020-06-19 14:35:33,666] DEBUG - dev-layout-fargate-agent | Completed flow run submission (id: ad6fc865-21c7-4d7a-8a48-dab73c51d8fb)
```

in the AWS Console I see the Task created, then goes to `PROVISIONING` then `PENDING`, then `DEPROVISIONING` and ends up with `STOPPED`.

Reason of the stop:
`Asm fetching username: AuthorizationData is malformed, empty field` :-(

Any suggestions how to debug it further?
@oleksandr I am not familiar enough with the AWS pattern here but maybe we can troubleshoot. Did you create your secret using the steps outlined here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth.html ?
@joshmeek true, silly me. I've updated the secret and passed the point of asm error. Now I've got container pending long time and then fails to start (`Task failed to start` in the reason), but I guess that's a different error and the scope of this issue is covered! 

Thanks a lot for the prompt fix! 
Will open a PR to add the fix!
@joshmeek I'm afraid we need to re-open this issue.

Here's the problem I was able to reproduce.

Passing credentials to private registry from environment (example from docker compose):

```yaml
    environment:
      ...
      - containerDefinitions_repositoryCredentials={"credentialsParameter":"arn:aws:secretsmanager:eu-central-1:*********:secret:**********"}
```

The logs show they should be picked up (`repositoryCredentials from environment variable`):

```
prefect-fargate-agent_1  | [2020-06-24 07:04:16,119] DEBUG - dev-layout-fargate-agent | Verbose logs enabled
prefect-fargate-agent_1  | [2020-06-24 07:04:16,120] DEBUG - dev-layout-fargate-agent | Environment variables: []
prefect-fargate-agent_1  | [2020-06-24 07:04:16,120] DEBUG - dev-layout-fargate-agent | Max polls: None
prefect-fargate-agent_1  | [2020-06-24 07:04:16,120] DEBUG - dev-layout-fargate-agent | Agent address:
prefect-fargate-agent_1  | [2020-06-24 07:04:16,120] DEBUG - dev-layout-fargate-agent | Log to Cloud: True
prefect-fargate-agent_1  | [2020-06-24 07:04:16,120] DEBUG - dev-layout-fargate-agent | Prefect backend: server
prefect-fargate-agent_1  | [2020-06-24 07:04:16,360] DEBUG - dev-layout-fargate-agent | cpu = 256
prefect-fargate-agent_1  | [2020-06-24 07:04:16,360] DEBUG - dev-layout-fargate-agent | memory = 512
prefect-fargate-agent_1  | [2020-06-24 07:04:16,360] DEBUG - dev-layout-fargate-agent | executionRoleArn = arn:aws:iam::*********:role/ecsTaskExecutionRole
prefect-fargate-agent_1  | [2020-06-24 07:04:16,360] DEBUG - dev-layout-fargate-agent | cluster = npn-ecs1
prefect-fargate-agent_1  | [2020-06-24 07:04:16,360] DEBUG - dev-layout-fargate-agent | networkConfiguration = {'awsvpcConfiguration': {'assignPublicIp': 'DISABLED', 'subnets': ['subnet-**********'], 'securityGroups': []}}
prefect-fargate-agent_1  | [2020-06-24 07:04:16,360] DEBUG - dev-layout-fargate-agent | Container definition: logConfiguration from environment variable
prefect-fargate-agent_1  | [2020-06-24 07:04:16,361] DEBUG - dev-layout-fargate-agent | Container definition: repositoryCredentials from environment variable
prefect-fargate-agent_1  | [2020-06-24 07:04:16,470] DEBUG - dev-layout-fargate-agent | Launch type: FARGATE
prefect-fargate-agent_1  | [2020-06-24 07:04:16,470] DEBUG - dev-layout-fargate-agent | Enable task revisions: False
prefect-fargate-agent_1  | [2020-06-24 07:04:16,470] DEBUG - dev-layout-fargate-agent | Use external kwargs: False
prefect-fargate-agent_1  | [2020-06-24 07:04:16,470] DEBUG - dev-layout-fargate-agent | External kwargs S3 bucket: None
prefect-fargate-agent_1  | [2020-06-24 07:04:16,470] DEBUG - dev-layout-fargate-agent | External kwargs S3 key: None
prefect-fargate-agent_1  |
...
```

Executing the flow triggers `taskDefinition` creation, which I can see in AWS Console but the Fargate container stoops immediately after provisioning/pending with an error:

```
CannotPullContainerError: Error response from daemon: Get https://registry.gitlab.com/v2/xxxxxxxx/sample-flow/manifests/master: denied: access forbidden
```

When I check `taskDefinition` created by Fargate agent in AWS Console (JSON Tab) the required section is missing, excepting `image` key.

This is what should be in definition (according to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth.html):

```json
// EXPECTED

"containerDefinitions": [
    {
        ...
        "image": "private-repo/private-image",
        "repositoryCredentials": {
            "credentialsParameter": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name"
        }
        ...
    }
]
```

This is what I see created upon flow run:

```json
// OBSERVED

"containerDefinitions": [
    {
       ...
        "image": "private-repo/private-image",
       ...
    }
]
```

Perhaps, there's an issue passing this parameter to boto3 client? On the opposite side I see that parameters for `logConfiguration` are passed properly.



Resolving patch:
diff --git a/src/prefect/agent/fargate/agent.py b/src/prefect/agent/fargate/agent.py
--- a/src/prefect/agent/fargate/agent.py
+++ b/src/prefect/agent/fargate/agent.py
@@ -618,6 +618,9 @@ def _create_task_definition(
         container_definitions[0]["logConfiguration"] = container_definitions_kwargs.get(
             "logConfiguration", {}
         )
+        container_definitions[0][
+            "repositoryCredentials"
+        ] = container_definitions_kwargs.get("repositoryCredentials", {})
 
         # Register task definition
         self.logger.debug(


Test patch:
diff --git a/tests/agent/test_fargate_agent.py b/tests/agent/test_fargate_agent.py
--- a/tests/agent/test_fargate_agent.py
+++ b/tests/agent/test_fargate_agent.py
@@ -900,6 +900,7 @@ def test_deploy_flow_register_task_definition_all_args(
                         "awslogs-create-group": "true",
                     },
                 },
+                "repositoryCredentials": ["repo"],
             }
         ],
     }
@@ -985,6 +986,7 @@ def test_deploy_flow_register_task_definition_all_args(
                     "awslogs-create-group": "true",
                 },
             },
+            "repositoryCredentials": ["repo"],
         }
     ]
     assert boto3_client.register_task_definition.call_args[1][
@@ -1100,6 +1102,7 @@ def test_deploy_flows_includes_agent_labels_in_environment(
             "secrets": [],
             "mountPoints": [],
             "logConfiguration": {},
+            "repositoryCredentials": {},
         }
     ]
     assert boto3_client.register_task_definition.call_args[1][
@@ -1169,6 +1172,7 @@ def test_deploy_flow_register_task_definition_no_repo_credentials(
             "secrets": [],
             "mountPoints": [],
             "logConfiguration": {},
+            "repositoryCredentials": {},
         }
     ]
 
@@ -1264,6 +1268,7 @@ def test_deploy_flows_enable_task_revisions_no_tags(
                 "secrets": [],
                 "mountPoints": [],
                 "logConfiguration": {},
+                "repositoryCredentials": {},
             }
         ],
         family="name",
@@ -1644,6 +1649,7 @@ def test_deploy_flows_enable_task_revisions_with_external_kwargs(
                 "secrets": [],
                 "mountPoints": [],
                 "logConfiguration": {},
+                "repositoryCredentials": {},
             }
         ],
         cpu="256",