Repository: ansible-collections/community.aws

Issue:
Add AWS ECS Capacity Provider Strategy Support
### Summary

Migrated from https://github.com/ansible/ansible/issues/67997 (it doesn't seem to have made it over automatically)

Add support for AWS ECS Cluster Capacity Provider Strategy configuration.

Additional note, I noticed this because I was creating a cluster to use with the GitLab CI Fargate driver and started getting `The platform version must be null when specifying an EC2 launch type.` when trying to launch a job. It worked with a manually created Cluster and Task Definition, so I looked closely and found the difference was the manually created cluster had two Capacity Providers and the Ansible created one had none, nor can you manually add them. It's clearly something the AWS UI takes care of, which you can do with the API (see additional info) but this module currently does not support. It means you can't really use it to set up a Fargate cluster at all.

### Issue Type

Feature Idea

### Component Name

ecs_cluster

### Additional Information

Enable configuration of ECS cluster capacity providers and strategies thereof.

```yaml
ecs_cluster:
  ...
  capacity_providers:
    - "FARGATE"
    - "FARGATE_SPOT"
  capacity_provider_strategy:
    - capacity_provider: "FARGATE"
      base: 1
      weight: 1
    - capacity_provider: "FARGATE_SPOT"
      weight: 100 
```

hashicorp/terraform-provider-aws#11150
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-capacity-providers.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.put_cluster_capacity_providers

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/ecs_cluster.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/ecs_cluster.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Java1Guy @jillr @markuman @s-hertel @tremble @wimnat
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
>  It means you can't really use it to set up a Fargate cluster at all.

hm I'm not sure about it.  
At work we create some fargate ecs clusters just with

```yml
- name: create ecs cluster
  ecs_cluster:
    name: serverless-housekeeping
    state: present
```

and we can run ecs taskdefinitions with `launch_type: FARGATE` without any problems in that cluster.

```yml
    - name: letsencrypt taskdefinition
      ecs_taskdefinition:
        family: letsencrypt
        cpu: "256"
        memory: "512"
        state: present
        network_mode: awsvpc
        launch_type: FARGATE
        execution_role_arn: "arn:aws:iam::{{ caller_facts.account }}:role/ecsTaskExecutionRole"
        task_role_arn: "arn:aws:iam::{{ caller_facts.account }}:role/letsencryptECSTask"
        region: eu-central-1
        containers:
          - name: letsencrypt
            environment:
              - name: KMS
                value: "{{ kms.ssm }}"
            essential: true
            image: "{{ caller_facts.account }}.dkr.ecr.eu-central-1.amazonaws.com/letsencrypt:latest"
            logConfiguration:
              logDriver: awslogs
              options:
                awslogs-group: /ecs/letsencrypt
                awslogs-region: eu-central-1
                awslogs-stream-prefix: ecs
      register: letsTD
```
>>    It means you can't really use it to set up a Fargate cluster at all.

> hm I'm not sure about it.

...which does not mean that the parameters should not be supported by `community.aws.ecs_cluster`  
I think its not that hard to implement.
Hi @markuman, thanks for the reply. That's interesting, I wonder what I'm doing wrong then? It's OT for the issue but for some reason my Ansible-created cluster can't launch Fargate task definitions but my manually created one *can* ... and I can't see any other difference. I'll keep digging though, if it works for you then at least I know it's possible! :-)
@gregharvey 

```yml
---
- hosts: localhost
  connection: local
  tags:
    - example

  vars:
    region: eu-central-1
    subnet: subnet-d8309db2
    security_group: sg-f32f0196
    ecs_trusted_relationship: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
              "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
        ]
      }


  tasks:
    - name: Get the current caller identity facts
      aws_caller_info:
      register: caller_facts

    - name: create ecsTaskExecution role
      iam_role:
        name: ecsTaskExecutionRole
        description: ecsTaskExecutionRole with to many permissions
        state: present
        purge_policies: yes
        managed_policy:
          - arn:aws:iam::aws:policy/CloudWatchLogsFullAccess
          - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
          - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
          - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceEventsRole
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        assume_role_policy_document: "{{ ecs_trusted_relationship }}"

    - name: create ecs cluster
      ecs_cluster:
        name: sometcluster
        state: present
        region: "{{ region }}"

    - name: create cloudwatch log group
      cloudwatchlogs_log_group:
        log_group_name: /ecs/fargate-test
        retention: 1
        region: "{{ region }}"

    - name: some fargate task definition
      ecs_taskdefinition:
        family: something
        cpu: "256"
        memory: "512"
        state: present
        network_mode: awsvpc
        launch_type: FARGATE
        execution_role_arn: ecsTaskExecutionRole
        task_role_arn: ecsTaskExecutionRole
        region: "{{ region }}"
        containers:
          - name: something
            command:
              - uptime
            essential: true
            image: "alpine:latest"
            logConfiguration:
              logDriver: awslogs
              options:
                awslogs-group: /ecs/fargate-test
                awslogs-region: "{{ region }}"
                awslogs-stream-prefix: ecs
      register: td_output

    - name: Run task
      community.aws.ecs_task:
        operation: run
        cluster: sometcluster
        task_definition: something
        count: 1
        started_by: ansible_user
        launch_type: FARGATE
        network_configuration:
          subnets:
            - "{{ subnet }}"
          security_groups:
            - "{{ security_group }}"
      register: task_output

    - debug:
        var: task_output

- hosts: localhost
  connection: local
  tags:
    - cleanup

  vars:
    region: eu-central-1
    subnet: subnet-d8309db2
    security_group: sg-f32f0196
    ecs_trusted_relationship: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
              "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
        ]
      }

  tasks:
    - name: remove iam role
      iam_role:
        name: ecsTaskExecutionRole
        description: ecsTaskExecutionRole with to many permissions
        state: absent
        purge_policies: yes
        managed_policy:
          - arn:aws:iam::aws:policy/CloudWatchLogsFullAccess
          - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
          - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
          - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceEventsRole
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        assume_role_policy_document: "{{ ecs_trusted_relationship }}"

    - name: remove ecs cluster
      ecs_cluster:
        name: sometcluster
        state: absent
        region: "{{ region }}"

    - name: remove cloudwatch log group
      cloudwatchlogs_log_group:
        log_group_name: /ecs/fargate-test
        retention: 1
        region: "{{ region }}"
        state: absent
```

adjust just the vars 

`AWS_PROFILE=yourprofile ansible-playbook 770.yml --tags example`  
and `AWS_PROFILE=yourprofile ansible-playbook 770.yml --tags cleanup` to remove the resources.

just the image is failing to pull (_no idea atm_)

```
 Stopped reason
CannotPullContainerError: inspect image has been retried 5 time(s): failed to resolve ref "docker.io/library/alpine:latest": failed to do request: Head https://registry-1.docker.io/v2/library/alpine/manifests/latest: dial tcp 52.204.76.244:443: i/o tim...
```

but at least, it works without any issue to run fargate container in a cluster made by `ecs_cluster` module.  
Thank you so much, I'll give it a go! :+1: 
Just to follow up here, in case someone has a similar problem. The code above works perfectly, so indeed you clearly *can* create a cluster and run a task. However, my GitLab `fargate` custom executor still wasn't working. I reviewed the docs to try and understand what's different, and for reasons I don't know there's steps 7 and 8 here to add a default capacity provider strategy:

* https://docs.gitlab.com/runner/configuration/runner_autoscale_aws_fargate/#step-5-create-an-ecs-fargate-cluster

Without that step it doesn't work. I presume the driver does not set the `launch_type` when it runs a task, and that it defaults to `EC2` if you don't either specify `FARGATE` when you launch the task *or* tell your cluster to favour `FARGATE`. This is really a bug in the `fargate` driver for GitLab Runner, in fairness, but I could work around it if Ansible let me set that default capacity provider strategy. So it would be handy. :-)

Resolving patch:
diff --git a/plugins/modules/ecs_cluster.py b/plugins/modules/ecs_cluster.py
--- a/plugins/modules/ecs_cluster.py
+++ b/plugins/modules/ecs_cluster.py
@@ -41,6 +41,41 @@
         required: false
         type: int
         default: 10
+    capacity_providers:
+        version_added: 5.2.0
+        description:
+            - List of capacity providers to use for the cluster.
+        required: false
+        type: list
+        elements: str
+    capacity_provider_strategy:
+        version_added: 5.2.0
+        description:
+            - List of capacity provider strategies to use for the cluster.
+        required: false
+        type: list
+        elements: dict
+        suboptions:
+            capacity_provider:
+                description:
+                  - Name of capacity provider.
+                type: str
+            weight:
+                description:
+                  - The relative percentage of the total number of launched tasks that should use the specified provider.
+                type: int
+            base:
+                description:
+                  - How many tasks, at a minimum, should use the specified provider.
+                type: int
+    purge_capacity_providers:
+        version_added: 5.2.0
+        description:
+            - Toggle overwriting of existing capacity providers or strategy. This is needed for backwards compatibility.
+            - By default I(purge_capacity_providers=false).  In a release after 2024-06-01 this will be changed to I(purge_capacity_providers=true).
+        required: false
+        type: bool
+        default: false
 extends_documentation_fragment:
 - amazon.aws.aws
 - amazon.aws.ec2
@@ -56,6 +91,21 @@
     name: default
     state: present
 
+- name: Cluster creation with capacity providers and strategies.
+  community.aws.ecs_cluster:
+    name: default
+    state: present
+    capacity_providers:
+      - FARGATE
+      - FARGATE_SPOT
+    capacity_provider_strategy:
+      - capacity_provider: FARGATE
+        base: 1
+        weight: 1
+      - capacity_provider: FARGATE_SPOT
+        weight: 100
+    purge_capacity_providers: True
+
 - name: Cluster deletion
   community.aws.ecs_cluster:
     name: default
@@ -75,6 +125,16 @@
     description: how many services are active in this cluster
     returned: 0 if a new cluster
     type: int
+capacityProviders:
+    version_added: 5.2.0
+    description: list of capacity providers used in this cluster
+    returned: always
+    type: list
+defaultCapacityProviderStrategy:
+    version_added: 5.2.0
+    description: list of capacity provider strategies used in this cluster
+    returned: always
+    type: list
 clusterArn:
     description: the ARN of the cluster just created
     type: str
@@ -112,6 +172,8 @@
     pass  # Handled by AnsibleAWSModule
 
 from ansible_collections.community.aws.plugins.module_utils.modules import AnsibleCommunityAWSModule as AnsibleAWSModule
+from ansible.module_utils.common.dict_transformations import snake_dict_to_camel_dict
+from ansible.module_utils.common.dict_transformations import camel_dict_to_snake_dict
 
 
 class EcsClusterManager:
@@ -145,8 +207,26 @@ def describe_cluster(self, cluster_name):
                 return c
         raise Exception("Unknown problem describing cluster %s." % cluster_name)
 
-    def create_cluster(self, clusterName='default'):
-        response = self.ecs.create_cluster(clusterName=clusterName)
+    def create_cluster(self, cluster_name, capacity_providers, capacity_provider_strategy):
+        params = dict(clusterName=cluster_name)
+        if capacity_providers:
+            params['capacityProviders'] = snake_dict_to_camel_dict(capacity_providers)
+        if capacity_provider_strategy:
+            params['defaultCapacityProviderStrategy'] = snake_dict_to_camel_dict(capacity_provider_strategy)
+        response = self.ecs.create_cluster(**params)
+        return response['cluster']
+
+    def update_cluster(self, cluster_name, capacity_providers, capacity_provider_strategy):
+        params = dict(cluster=cluster_name)
+        if capacity_providers:
+            params['capacityProviders'] = snake_dict_to_camel_dict(capacity_providers)
+        else:
+            params['capacityProviders'] = []
+        if capacity_provider_strategy:
+            params['defaultCapacityProviderStrategy'] = snake_dict_to_camel_dict(capacity_provider_strategy)
+        else:
+            params['defaultCapacityProviderStrategy'] = []
+        response = self.ecs.put_cluster_capacity_providers(**params)
         return response['cluster']
 
     def delete_cluster(self, clusterName):
@@ -159,7 +239,17 @@ def main():
         state=dict(required=True, choices=['present', 'absent', 'has_instances']),
         name=dict(required=True, type='str'),
         delay=dict(required=False, type='int', default=10),
-        repeat=dict(required=False, type='int', default=10)
+        repeat=dict(required=False, type='int', default=10),
+        purge_capacity_providers=dict(required=False, type='bool', default=False),
+        capacity_providers=dict(required=False, type='list', elements='str'),
+        capacity_provider_strategy=dict(required=False,
+                                        type='list',
+                                        elements='dict',
+                                        options=dict(capacity_provider=dict(type='str'),
+                                                     weight=dict(type='int'),
+                                                     base=dict(type='int', default=0)
+                                                     )
+                                        ),
     )
     required_together = [['state', 'name']]
 
@@ -177,12 +267,53 @@ def main():
 
     results = dict(changed=False)
     if module.params['state'] == 'present':
+        # Pull requested and existing capacity providers and strategies.
+        purge_capacity_providers = module.params['purge_capacity_providers']
+        requested_cp = module.params['capacity_providers']
+        requested_cps = module.params['capacity_provider_strategy']
         if existing and 'status' in existing and existing['status'] == "ACTIVE":
-            results['cluster'] = existing
+            existing_cp = existing['capacityProviders']
+            existing_cps = existing['defaultCapacityProviderStrategy']
+
+            if requested_cp is None:
+                requested_cp = []
+
+            # Check if capacity provider strategy needs to trigger an update.
+            cps_update_needed = False
+            if requested_cps is not None:
+                for strategy in requested_cps:
+                    if snake_dict_to_camel_dict(strategy) not in existing_cps:
+                        cps_update_needed = True
+                for strategy in existing_cps:
+                    if camel_dict_to_snake_dict(strategy) not in requested_cps:
+                        cps_update_needed = True
+            elif requested_cps is None and existing_cps != []:
+                cps_update_needed = True
+
+            # Unless purge_capacity_providers is true, we will not be updating the providers or strategy.
+            if not purge_capacity_providers:
+                module.deprecate('After 2024-06-01 the default value of purge_capacity_providers will change from false to true.'
+                                 ' To maintain the existing behaviour explicitly set purge_capacity_providers=true',
+                                 date='2024-06-01', collection_name='community.aws')
+                cps_update_needed = False
+                requested_cp = existing_cp
+                requested_cps = existing_cps
+
+            # If either the providers or strategy differ, update the cluster.
+            if requested_cp != existing_cp or cps_update_needed:
+                if not module.check_mode:
+                    results['cluster'] = cluster_mgr.update_cluster(cluster_name=module.params['name'],
+                                                                    capacity_providers=requested_cp,
+                                                                    capacity_provider_strategy=requested_cps)
+                results['changed'] = True
+            else:
+                results['cluster'] = existing
         else:
             if not module.check_mode:
                 # doesn't exist. create it.
-                results['cluster'] = cluster_mgr.create_cluster(module.params['name'])
+                results['cluster'] = cluster_mgr.create_cluster(cluster_name=module.params['name'],
+                                                                capacity_providers=requested_cp,
+                                                                capacity_provider_strategy=requested_cps)
             results['changed'] = True
 
     # delete the cluster


Test patch:
diff --git a/tests/integration/targets/ecs_cluster/tasks/main.yml b/tests/integration/targets/ecs_cluster/tasks/main.yml
--- a/tests/integration/targets/ecs_cluster/tasks/main.yml
+++ b/tests/integration/targets/ecs_cluster/tasks/main.yml
@@ -63,6 +63,30 @@
         that:
           - not ecs_cluster_again.changed
 
+    - name: add capacity providers and strategy
+      ecs_cluster:
+        name: "{{ ecs_cluster_name }}"
+        state: present
+        purge_capacity_providers: True
+        capacity_providers:
+          - FARGATE
+          - FARGATE_SPOT
+        capacity_provider_strategy:
+          - capacity_provider: FARGATE
+            base: 1
+            weight: 1
+          - capacity_provider: FARGATE_SPOT
+            weight: 100
+      register: ecs_cluster_update
+
+    - name: check that ecs_cluster was correctly updated
+      assert:
+        that:
+          - ecs_cluster_update.changed
+          - ecs_cluster_update.cluster is defined
+          - ecs_cluster_update.cluster.capacityProviders is defined
+          - "'FARGATE' in ecs_cluster_update.cluster.capacityProviders"
+
     - name: create a VPC to work in
       ec2_vpc_net:
         cidr_block: 10.0.0.0/16