Repository: ansible-collections/community.aws

Issue:
ecs_service cannot handle distinctInstance Placement Constraint
### Summary

Most Placement Constraint options have both a `type` and an `expression`.  You can see this is `ecs_service.py` `main()` ...

```
        placement_constraints=dict(
            required=False,
            default=[],
            type='list',
            elements='dict',
            options=dict(
                type=dict(type='str'),
                expression=dict(type='str')
            )
        ),
```

However, the `distinctInstance` Placement Constraint does not require an `expression.`  The bug here is that the above code effectively mandates that `expression` is required.  If it is omitted from the playbook, we end up with the value being `None` which I was able to capture by modifying the code ....

```
[{'type': 'distinctInstance', 'expression': None}] 
```

The presence of `expression` here, even with the value of `None`, yields the following error:

```
botocore.exceptions.ParamValidationError: Parameter validation failed:\nInvalid type for parameter placementConstraints[0].expression, value: None,
```

I've confirmed that `boto` supports this just fine ...

```
params = dict(
    cluster="default",
    serviceName="some-service-name",
    taskDefinition="some-task-definition,
    desiredCount=0,
    placementConstraints=[dict(type="distinctInstance")]
)

print("params:" + str(params))
ecs.create_service(**params) 
```

The bug appears to be in `ecs_service.py` during the **create service** path of execution. 


### Issue Type

Bug Report

### Component Name

ecs_service

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.12.4]
  config file = None
  configured module search path = ['/Users/wetorres/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/wetorres/Library/Python/3.8/lib/python/site-packages/ansible
  ansible collection location = /Users/wetorres/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/wetorres/Library/Python/3.8/bin/ansible
  python version = 3.8.9 (default, Feb 18 2022, 07:45:34) [Clang 13.1.6 (clang-1316.0.21.2)]
  jinja version = 3.1.1
  libyaml = True
```


### Collection Versions

```console (paste below)
$ ansible-galaxy collection list

# /Users/wetorres/Library/Python/3.8/lib/python/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.5.1  
ansible.posix                 1.3.0  
ansible.utils                 2.5.2  
ansible.windows               1.9.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.11.0 
check_point.mgmt              2.3.0  
chocolatey.chocolatey         1.2.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.18 
cisco.ios                     2.8.0  
cisco.iosxr                   2.8.1  
cisco.ise                     1.2.1  
cisco.meraki                  2.6.0  
cisco.mso                     1.3.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.9.0  
cisco.ucs                     1.7.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.3.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.2.3  
community.digitalocean        1.15.1 
community.dns                 2.0.8  
community.docker              2.2.1  
community.fortios             1.0.0  
community.general             4.6.0  
community.google              1.0.0  
community.grafana             1.3.3  
community.hashi_vault         2.3.0  
community.hrobot              1.2.2  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.5  
community.network             3.1.0  
community.okd                 2.1.0  
community.postgresql          1.7.1  
community.proxysql            1.3.1  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.17.1 
community.windows             1.9.0  
community.zabbix              1.5.1  
containers.podman             1.9.1  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.4.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.15.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.4  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.4  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.3  
infoblox.nios_modules         1.2.1  
inspur.sm                     1.3.0  
junipernetworks.junos         2.9.0  
kubernetes.core               2.2.3  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.15.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.17.3
netapp.storagegrid            21.9.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.6.0  
ngine_io.cloudstack           2.2.3  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.7.2  
openvswitch.openvswitch       2.1.0  
ovirt.ovirt                   1.6.6  
purestorage.flasharray        1.12.1 
purestorage.flashblade        1.9.0  
sensu.sensu_go                1.13.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.27.1 
theforeman.foreman            2.2.0  
vyos.vyos                     2.8.0  
wti.remote                    1.0.3  

# /Users/wetorres/.ansible/collections/ansible_collections
Collection Version
---------- -------
amazon.aws 3.1.1  
```


### AWS SDK versions

```console (paste below)
$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.21.29
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/wetorres/Library/Python/3.8/lib/python/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.24.29
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/wetorres/Library/Python/3.8/lib/python/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer
```

We do a lot of our ansible stuff from with virtual envs and docker containers.  I don't happen to have `boto` installed, but don't let that be a distraction.  The bug here is in `ecs_service.py`.


### Configuration

```console (paste below)
$ ansible-config dump --only-changed

```
I ran this and it produced no output

### OS / Environment

```
uname -a
Darwin REDACTED 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:45:05 PDT 2022; root:xnu-8020.101.4~15/RELEASE_X86_64 x86_64
```


### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
---
- name: "ecs_service_bug"
  hosts: localhost

  tasks:
  - name: "ecs_service_bug"
    ecs_service:
      state: present
      name: "ecs_service_bug"
      cluster: "default"
      desired_count: 0
      region: "us-east-1"
      task_definition: "any-valid-task-definition"
      force_new_deployment: "yes"
      placement_constraints:
        - type: distinctInstance 
```

Run `ansible-playbook ./this.yml` to reproduce the error.  

Notes:
- **If the ECS Service does not exist**, running this playbook will generate the error above.  
- **If the ECS Service already exists**, no error will be produced 
    - **If the existing ECS Service** has the `distinctInstance` Placement Constraint, it will be retained.
    - **If the existing ECS Service** does not have the `distinctInstance` Placement Constraint, it will not be added.



### Expected Results

- I expect to be able to define `distinctInstance` Placement Strategy in my Ansible playbook, regardless of the state of the ECS Service. 
- I expect the `distinctInstance` Placement Strategy defined in the playbok should be applied to the ECS Service in question. 

### Actual Results



1. If the ECS Service doesn't exist yet, `ansible-playbook` generates an error and fails.  No ECS Service is created.
2. If the ECS Service exists _with_ `distinctInstance` Placement Strategy, this Placement Strategy is retained.
3. If the ECS Service exists _without_ `distinctInstance` Placement Strategy, this Placement Strategy (which is supplied in the playbook) is _not_ applied to the ECS Service.


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/ecs_service.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/ecs_service.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Java1Guy @jillr @kaczynskid @markuman @s-hertel @tremble @zacblazic
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@superwesman  
Thx for the detail reporting. I can confirm the bug.

> The bug here is that the above code effectively mandates that `expression` is required.

But that's not valid. The parameter is set to `required=False,`.  
The parameter `placement_constraints`  is just path through, from [789](https://github.com/ansible-collections/community.aws/blob/c403552f8d8d2da6be0a1797d6ac85ab1cd7b22c/plugins/modules/ecs_service.py#L789)  to [Line 599](https://github.com/ansible-collections/community.aws/blob/c403552f8d8d2da6be0a1797d6ac85ab1cd7b22c/plugins/modules/ecs_service.py#L599).

A possible fix might be easy. All what's to do is to pop `None` values from `params['placement_constraints']` somewhere [here](https://github.com/ansible-collections/community.aws/blob/c403552f8d8d2da6be0a1797d6ac85ab1cd7b22c/plugins/modules/ecs_service.py#L583-L598) ...
* plus some integration test to cover that case
* plus a changelog fragment

@superwesman are you willing to provide a PR to fix this issue?

hello @markuman - Unfortunately, I don't think I'm in a position to provide a PR.  I've discovered this bug at work and the exact corporate policies related to open source development are many, and not clear to me.  I hope that my input is at least helpful for anyone who does have capacity to implement.  I'm sorry I can't do more at the moment.
@markuman I think I can provide a PR. I have been investigating issues with our automation and ended up having a [solution](https://github.com/ansible-collections/community.aws/compare/main...novak-as:main) which looks pretty similar to your proposal. I can confirm that this worked for me, so it looks like not much work is left
> @markuman I think I can provide a PR. I have been investigating issues with our automation and ended up having a [solution](https://github.com/ansible-collections/community.aws/compare/main...novak-as:main) which looks pretty similar to your proposal. I can confirm that this worked for me, so it looks like not much work is left

That would be awesome @novak-as  
I've test your branch and the current integration test passes. From my perspective, there are 3 steps left.

1. [rebase your branch with upstream main branch](https://docs.ansible.com/ansible/latest/dev_guide/developing_rebasing.html)
2. open PR and add a bugfixes changelog fragment
3.  expand the integration test that covers the case

just a few days left until we're going to release 3.3.0. But I think we can made it.

Resolving patch:
diff --git a/plugins/modules/ecs_service.py b/plugins/modules/ecs_service.py
--- a/plugins/modules/ecs_service.py
+++ b/plugins/modules/ecs_service.py
@@ -113,6 +113,7 @@
             type: str
           expression:
             description: A cluster query language expression to apply to the constraint.
+            required: false
             type: str
     placement_strategy:
         description:
@@ -584,7 +585,6 @@ def create_service(self, service_name, cluster_name, task_definition, load_balan
             clientToken=client_token,
             role=role,
             deploymentConfiguration=deployment_configuration,
-            placementConstraints=placement_constraints,
             placementStrategy=placement_strategy
         )
         if network_configuration:
@@ -597,6 +597,13 @@ def create_service(self, service_name, cluster_name, task_definition, load_balan
             params['healthCheckGracePeriodSeconds'] = health_check_grace_period_seconds
         if service_registries:
             params['serviceRegistries'] = service_registries
+
+        # filter placement_constraint and left only those where value is not None
+        # use-case: `distinctInstance` type should never contain `expression`, but None will fail `str` type validation
+        if placement_constraints:
+            params['placementConstraints'] = [{key: value for key, value in constraint.items() if value is not None}
+                                              for constraint in placement_constraints]
+
         # desired count is not required if scheduling strategy is daemon
         if desired_count is not None:
             params['desiredCount'] = desired_count
@@ -674,7 +681,7 @@ def main():
             elements='dict',
             options=dict(
                 type=dict(type='str'),
-                expression=dict(type='str')
+                expression=dict(required=False, type='str')
             )
         ),
         placement_strategy=dict(


Test patch:
diff --git a/tests/integration/targets/ecs_cluster/tasks/main.yml b/tests/integration/targets/ecs_cluster/tasks/main.yml
--- a/tests/integration/targets/ecs_cluster/tasks/main.yml
+++ b/tests/integration/targets/ecs_cluster/tasks/main.yml
@@ -215,6 +215,8 @@
         desired_count: 1
         deployment_configuration: "{{ ecs_service_deployment_configuration }}"
         placement_strategy: "{{ ecs_service_placement_strategy }}"
+        placement_constraints:
+          - type: distinctInstance
         health_check_grace_period_seconds: "{{ ecs_service_health_check_grace_period }}"
         load_balancers:
           - targetGroupArn: "{{ elb_target_group_instance.target_group_arn }}"
@@ -223,6 +225,12 @@
         role: "ecsServiceRole"
       register: ecs_service
 
+    - name: check that placement constraint has been applied
+      assert:
+        that:
+          - ecs_service.changed
+          - "ecs_service.service.placementConstraints[0].type == 'distinctInstance'"
+
     - name: check that ECS service creation changed
       assert:
         that:
@@ -237,6 +245,8 @@
         desired_count: 1
         deployment_configuration: "{{ ecs_service_deployment_configuration }}"
         placement_strategy: "{{ ecs_service_placement_strategy }}"
+        placement_constraints:
+          - type: distinctInstance
         health_check_grace_period_seconds: "{{ ecs_service_health_check_grace_period }}"
         load_balancers:
           - targetGroupArn: "{{ elb_target_group_instance.target_group_arn }}"
@@ -260,6 +270,8 @@
         desired_count: 1
         deployment_configuration: "{{ ecs_service_deployment_configuration }}"
         placement_strategy: "{{ ecs_service_placement_strategy }}"
+        placement_constraints:
+          - type: distinctInstance
         health_check_grace_period_seconds: "{{ ecs_service_health_check_grace_period }}"
         load_balancers:
           - targetGroupArn: "{{ elb_target_group_instance.target_group_arn }}"