Repository: ansible-collections/community.aws

Issue:
elb_application_lb with empty security groups list behaves inconsistently on create/update
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->
Resubmitted from https://github.com/ansible-collections/amazon.aws/issues/10

##### SUMMARY
<!--- Explain the problem briefly below -->
`elb_application_lb` requires the `security_groups` option when `state=present` as explained in the docs (although it also says that the default is `[]` which seems useless since it won't accept the option being omitted).

When creating a new ALB and supplying `security_groups: []` explicitly, the ALB is created successfully with the VPC default SG.

Running the same task again will fail with the error that the `security_groups` option is missing,

I'm not sure if this is reproducible outside of a VPC since I'm not sure there is such a thing as a default SG in that case.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
elb_application_lb

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
2.9.6

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->


##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
    - elb_application_lb:
        region: "us-east-1"
        name: "repro-delete"
        state: "present"
        subnets: "{{ my_subnets }}"
        listeners:
          - Protocol: HTTP
            Port: 80
            DefaultActions:
              - Type: forward
                TargetGroupName: repro-group-us-east-1a
        scheme: internal
        security_groups: []
        wait: yes
      register: alb
      loop: [1, 2]
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
ALB is created, then second run is `ok`.

(an acceptable result might also be that the first run fails with an invalid option value, but that does preclude the possibility of using a "default" SG)

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
Second run fails.

<!--- Paste verbatim command output between quotes -->
```paste below
fatal: [localhost]: FAILED! => {"changed": false, "msg": "state is present but all of the following are missing: security_groups"}
```



Discussion:
Files identified in the description:
* [`plugins/modules/elb_application_lb.py`](https://github.com/ansible-collections/community.aws/blob/main/plugins/modules/elb_application_lb.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @jillr @s-hertel @tremble @wimnat
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@briantist Thank you for reporting this! We have recently this module to our CI so it should be easier to add a test case for this.
> I'm not sure if this is reproducible outside of a VPC since I'm not sure there is such a thing as a default SG in that case.

imo you can create ALB resources only **in** a VPC.  

the only way to solve this issue is imo to not allow an empty array as an input for `security_groups` parameter.  
otherwise and empty array must be treated as "use the VPCs default security group", which must be determined inside the code.
I believe that "use the default VPC SG" is a desirable option in some/many use cases, so I wouldn't want to see that option disappear. It would be most useful if the module handled that case correctly, even if internally it means it has to retrieve the VPC default in order to verify its state.
I can work on this @briantist @markuman. Expected behavior is treat empty array as "use the VPCs default security group" ?
> I can work on this @briantist @markuman. Expected behavior is treat empty array as "use the VPCs default security group" ?

@jatorcasso kind of yes, that's already what happens on first creation; the problem is that running it again will fail. So when the ALB already exists, the module needs to have a way to verify the current state, and that part is not working now. It may require reading the default SG from the VPC in order to verify.

It might also be desirable to have "use the default VPC" as a separate option and then disallow the empty SG list, not sure...

I might also suggest that no security group option supplied, when the ALB exists, might mean "don't check or change the state of SGs" but that is a behavioral change.
If that's what happens on first creation, then that's what should happen after first creation as well imo to maintain idempotence. I'll work on that and add some integration tests to validate

Resolving patch:
diff --git a/plugins/modules/elb_application_lb.py b/plugins/modules/elb_application_lb.py
--- a/plugins/modules/elb_application_lb.py
+++ b/plugins/modules/elb_application_lb.py
@@ -144,7 +144,7 @@
     description:
       - A list of the names or IDs of the security groups to assign to the load balancer.
       - Required if I(state=present).
-    default: []
+      - If C([]), the VPC's default security group will be used.
     type: list
     elements: str
   scheme:
@@ -494,10 +494,16 @@
     type: bool
     sample: false
 '''
+try:
+    import botocore
+except ImportError:
+    pass  # caught by AnsibleAWSModule
 
 from ansible_collections.amazon.aws.plugins.module_utils.core import AnsibleAWSModule
-from ansible_collections.amazon.aws.plugins.module_utils.ec2 import camel_dict_to_snake_dict
+from ansible_collections.amazon.aws.plugins.module_utils.ec2 import ansible_dict_to_boto3_filter_list
+from ansible_collections.amazon.aws.plugins.module_utils.ec2 import AWSRetry
 from ansible_collections.amazon.aws.plugins.module_utils.ec2 import boto3_tag_list_to_ansible_dict
+from ansible_collections.amazon.aws.plugins.module_utils.ec2 import camel_dict_to_snake_dict
 from ansible_collections.amazon.aws.plugins.module_utils.ec2 import compare_aws_tags
 from ansible_collections.amazon.aws.plugins.module_utils.elbv2 import (
     ApplicationLoadBalancer,
@@ -509,6 +515,29 @@
 from ansible_collections.amazon.aws.plugins.module_utils.elb_utils import get_elb_listener_rules
 
 
+@AWSRetry.jittered_backoff()
+def describe_sgs_with_backoff(connection, **params):
+    paginator = connection.get_paginator('describe_security_groups')
+    return paginator.paginate(**params).build_full_result()['SecurityGroups']
+
+
+def find_default_sg(connection, module, vpc_id):
+    """
+    Finds the default security group for the given VPC ID.
+    """
+    filters = ansible_dict_to_boto3_filter_list({'vpc-id': vpc_id, 'group-name': 'default'})
+    try:
+        sg = describe_sgs_with_backoff(connection, Filters=filters)
+    except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
+        module.fail_json_aws(e, msg='No default security group found for VPC {0}'.format(vpc_id))
+    if len(sg) == 1:
+        return sg[0]['GroupId']
+    elif len(sg) == 0:
+        module.fail_json(msg='No default security group found for VPC {0}'.format(vpc_id))
+    else:
+        module.fail_json(msg='Multiple security groups named "default" found for VPC {0}'.format(vpc_id))
+
+
 def create_or_update_alb(alb_obj):
     """Create ALB or modify main attributes. json_exit here"""
     if alb_obj.elb:
@@ -738,6 +767,11 @@ def main():
 
     alb = ApplicationLoadBalancer(connection, connection_ec2, module)
 
+    # Update security group if default is specified
+    if alb.elb and module.params.get('security_groups') == []:
+        module.params['security_groups'] = [find_default_sg(connection_ec2, module, alb.elb['VpcId'])]
+        alb = ApplicationLoadBalancer(connection, connection_ec2, module)
+
     if state == 'present':
         create_or_update_alb(alb)
     elif state == 'absent':


Test patch:
diff --git a/tests/integration/targets/elb_application_lb/tasks/main.yml b/tests/integration/targets/elb_application_lb/tasks/main.yml
--- a/tests/integration/targets/elb_application_lb/tasks/main.yml
+++ b/tests/integration/targets/elb_application_lb/tasks/main.yml
@@ -25,6 +25,12 @@
       set_fact:
         vpc_id: "{{ vpc.vpc.id }}"
 
+    - name: Get VPC's default security group
+      ec2_group_info:
+        filters:
+          vpc-id: "{{ vpc_id }}"
+      register: default_sg
+
     - name: Create an internet gateway
       ec2_vpc_igw:
         vpc_id: '{{ vpc_id }}'
@@ -200,7 +206,90 @@
 
     # ------------------------------------------------------------------------------------------
 
-    - name: Create an ALB with ip address type - check_mode
+    - name: Create an ALB with defaults - check_mode
+      elb_application_lb:
+        name: "{{ alb_name }}"
+        subnets: "{{ public_subnets }}"
+        security_groups: []
+        state: present
+        listeners:
+          - Protocol: HTTP
+            Port: 80
+            DefaultActions:
+              - Type: forward
+                TargetGroupName: "{{ tg_name }}"
+      register: alb
+      check_mode: yes
+
+    - assert:
+        that:
+          - alb is changed
+          - alb.msg is match('Would have created ALB if not in check mode.')
+
+    - name: Create an ALB with defaults
+      elb_application_lb:
+        name: "{{ alb_name }}"
+        subnets: "{{ public_subnets }}"
+        security_groups: []
+        state: present
+        listeners:
+          - Protocol: HTTP
+            Port: 80
+            DefaultActions:
+              - Type: forward
+                TargetGroupName: "{{ tg_name }}"
+      register: alb
+
+    - assert:
+        that:
+          - alb is changed
+          - alb.listeners[0].rules | length == 1
+          - alb.security_groups | length == 1
+          - alb.security_groups[0] == default_sg.security_groups[0].group_id
+
+    - name: Create an ALB with defaults (idempotence) - check_mode
+      elb_application_lb:
+        name: "{{ alb_name }}"
+        subnets: "{{ public_subnets }}"
+        security_groups: []
+        state: present
+        listeners:
+          - Protocol: HTTP
+            Port: 80
+            DefaultActions:
+              - Type: forward
+                TargetGroupName: "{{ tg_name }}"
+      register: alb
+      check_mode: yes
+
+    - assert:
+        that:
+          - alb is not changed
+          - alb.msg is match('IN CHECK MODE - no changes to make to ALB specified.')
+
+    - name: Create an ALB with defaults (idempotence)
+      elb_application_lb:
+        name: "{{ alb_name }}"
+        subnets: "{{ public_subnets }}"
+        security_groups: []
+        state: present
+        listeners:
+          - Protocol: HTTP
+            Port: 80
+            DefaultActions:
+              - Type: forward
+                TargetGroupName: "{{ tg_name }}"
+      register: alb
+
+    - assert:
+        that:
+          - alb is not changed
+          - alb.listeners[0].rules | length == 1
+          - alb.security_groups[0] == default_sg.security_groups[0].group_id
+
+    # ------------------------------------------------------------------------------------------
+
+    - name: Update an ALB with ip address type - check_mode
       elb_application_lb:
         name: "{{ alb_name }}"
         subnets: "{{ public_subnets }}"
@@ -219,9 +308,9 @@
     - assert:
         that:
           - alb is changed
-          - alb.msg is match('Would have created ALB if not in check mode.')
+          - alb.msg is match('Would have updated ALB if not in check mode.')
 
-    - name: Create an ALB with ip address type
+    - name: Update an ALB with ip address type
       elb_application_lb:
         name: "{{ alb_name }}"
         subnets: "{{ public_subnets }}"