Repository: ansible/ansible

Issue:
ec2_group Idempotence
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
ec2_group

##### ANSIBLE VERSION
```
ansible 2.3.0.0
  config file = /home/con5cience/git/Ansible/ansible.cfg
  configured module search path = [u'/usr/share/ansible', u'library']
  python version = 2.7.13 (default, Jan 12 2017, 17:59:37) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
```

##### CONFIGURATION
```
[defaults]
inventory=inventory
library=/usr/share/ansible:library
roles_path=roles
vault_password_file = vault_pass
forks=20
ask_sudo_pass=yes
nocows=0
cow_selection = random
host_key_checking = False

[ssh_connection]
pipelining=True
ssh_args = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=30m
control_path = ~/.ssh/ansible-%%r@%%h:%%p
```

##### OS / ENVIRONMENT
From: Fedora 25
To: Amazon EC2 API

##### SUMMARY
Managing the rules of EC2 security groups using the `ec2_group` module is not idempotent.

##### STEPS TO REPRODUCE
1) Use Ansible to configure a security group and add rules to the group.
2) Run Ansible again.

<!--- Paste example playbooks or commands between quotes below -->
```yaml
      ec2_group:
        name: "{{ security_group_name|default( inventory_hostname ~ '-' ~ env ~ '-security-group' ) }}"
        description: "{{ security_group_name|default( inventory_hostname ~ '-' ~ env ~ 'Security Group' ) }}"
        vpc_id: "{{ ec2_vpc_id }}"
        region: "{{ ec2_region }}"
        rules: "{{ common_rules_ingress + rules_ingress }}"
        rules_egress: "{{ common_rules_egress + rules_egress }}"
      delegate_to: localhost
```
Where `common_rules_ingress`, `common_rules_egress`, `rules_ingress`, and `rules_egress` are lists of port rules in the style of:
```yaml
  - proto: tcp
    from_port: 443
    to_port: 443
    cidr_ip: 0.0.0.0/0

  - proto: tcp
    from_port: 80
    to_port: 80
    cidr_ip: 0.0.0.0/0
```

##### EXPECTED RESULTS
Task exits with 0 and playbook continues if the rule already exists and no changes are made.

##### ACTUAL RESULTS
Task exits with 1 (or 2, not entirely sure) and playbook run immediately bails when a rule already exists and no changes are made.
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_em0SmM/ansible_module_ec2_group.py", line 487, in <module>
    main()
  File "/tmp/ansible_em0SmM/ansible_module_ec2_group.py", line 439, in main
    cidr_ip=thisip)
  File "/usr/lib/python2.7/site-packages/boto/ec2/connection.py", line 3245, in authorize_security_group_egress
    params, verb='POST')
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule "peer: 0.0.0.0/0, TCP, from port: 443, to port: 443, ALLOW" already exists</Message></Error></Errors><RequestID>871be7ab-957c-4c41-8982-706e7a5dc64c</RequestID></Response>

fatal: [remotehost -> localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_em0SmM/ansible_module_ec2_group.py\", line 487, in <module>\n    main()\n  File \"/tmp/ansible_em0SmM/ansible_module_ec2_group.py\", line 439, in main\n    cidr_ip=thisip)\n  File \"/usr/lib/python2.7/site-packages/boto/ec2/connection.py\", line 3245, in authorize_security_group_egress\n    params, verb='POST')\n  File \"/usr/lib/python2.7/site-packages/boto/connection.py\", line 1227, in get_status\n    raise self.ResponseError(response.status, response.reason, body)\nboto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule \"peer: 0.0.0.0/0, TCP, from port: 443, to port: 443, ALLOW\" already exists</Message></Error></Errors><RequestID>871be7ab-957c-4c41-8982-706e7a5dc64c</RequestID></Response>\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
```

See also:
https://github.com/ansible/ansible/issues/5643 (arbitrarily closed)
https://github.com/ansible/ansible-modules-core/issues/2576 (open but in locked repo)
ec2_group Idempotence
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
ec2_group

##### ANSIBLE VERSION
```
ansible 2.3.0.0
  config file = /home/con5cience/git/Ansible/ansible.cfg
  configured module search path = [u'/usr/share/ansible', u'library']
  python version = 2.7.13 (default, Jan 12 2017, 17:59:37) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
```

##### CONFIGURATION
```
[defaults]
inventory=inventory
library=/usr/share/ansible:library
roles_path=roles
vault_password_file = vault_pass
forks=20
ask_sudo_pass=yes
nocows=0
cow_selection = random
host_key_checking = False

[ssh_connection]
pipelining=True
ssh_args = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=30m
control_path = ~/.ssh/ansible-%%r@%%h:%%p
```

##### OS / ENVIRONMENT
From: Fedora 25
To: Amazon EC2 API

##### SUMMARY
Managing the rules of EC2 security groups using the `ec2_group` module is not idempotent.

##### STEPS TO REPRODUCE
1) Use Ansible to configure a security group and add rules to the group.
2) Run Ansible again.

<!--- Paste example playbooks or commands between quotes below -->
```yaml
      ec2_group:
        name: "{{ security_group_name|default( inventory_hostname ~ '-' ~ env ~ '-security-group' ) }}"
        description: "{{ security_group_name|default( inventory_hostname ~ '-' ~ env ~ 'Security Group' ) }}"
        vpc_id: "{{ ec2_vpc_id }}"
        region: "{{ ec2_region }}"
        rules: "{{ common_rules_ingress + rules_ingress }}"
        rules_egress: "{{ common_rules_egress + rules_egress }}"
      delegate_to: localhost
```
Where `common_rules_ingress`, `common_rules_egress`, `rules_ingress`, and `rules_egress` are lists of port rules in the style of:
```yaml
  - proto: tcp
    from_port: 443
    to_port: 443
    cidr_ip: 0.0.0.0/0

  - proto: tcp
    from_port: 80
    to_port: 80
    cidr_ip: 0.0.0.0/0
```

##### EXPECTED RESULTS
Task exits with 0 and playbook continues if the rule already exists and no changes are made.

##### ACTUAL RESULTS
Task exits with 1 (or 2, not entirely sure) and playbook run immediately bails when a rule already exists and no changes are made.
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_em0SmM/ansible_module_ec2_group.py", line 487, in <module>
    main()
  File "/tmp/ansible_em0SmM/ansible_module_ec2_group.py", line 439, in main
    cidr_ip=thisip)
  File "/usr/lib/python2.7/site-packages/boto/ec2/connection.py", line 3245, in authorize_security_group_egress
    params, verb='POST')
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule "peer: 0.0.0.0/0, TCP, from port: 443, to port: 443, ALLOW" already exists</Message></Error></Errors><RequestID>871be7ab-957c-4c41-8982-706e7a5dc64c</RequestID></Response>

fatal: [remotehost -> localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_em0SmM/ansible_module_ec2_group.py\", line 487, in <module>\n    main()\n  File \"/tmp/ansible_em0SmM/ansible_module_ec2_group.py\", line 439, in main\n    cidr_ip=thisip)\n  File \"/usr/lib/python2.7/site-packages/boto/ec2/connection.py\", line 3245, in authorize_security_group_egress\n    params, verb='POST')\n  File \"/usr/lib/python2.7/site-packages/boto/connection.py\", line 1227, in get_status\n    raise self.ResponseError(response.status, response.reason, body)\nboto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule \"peer: 0.0.0.0/0, TCP, from port: 443, to port: 443, ALLOW\" already exists</Message></Error></Errors><RequestID>871be7ab-957c-4c41-8982-706e7a5dc64c</RequestID></Response>\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
```

See also:
https://github.com/ansible/ansible/issues/5643 (arbitrarily closed)
https://github.com/ansible/ansible-modules-core/issues/2576 (open but in locked repo)


Discussion:

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2_group.py b/lib/ansible/modules/cloud/amazon/ec2_group.py
--- a/lib/ansible/modules/cloud/amazon/ec2_group.py
+++ b/lib/ansible/modules/cloud/amazon/ec2_group.py
@@ -171,6 +171,7 @@
           - sg-edcd9784
 '''
 
+import json
 import re
 import time
 from ansible.module_utils.basic import AnsibleModule
@@ -187,6 +188,13 @@
 import traceback
 
 
+def deduplicate_rules_args(rules):
+    """Returns unique rules"""
+    if rules is None:
+        return None
+    return list(dict(zip((json.dumps(r, sort_keys=True) for r in rules), rules)).values())
+
+
 def make_rule_key(prefix, rule, group_id, cidr_ip):
     """Creates a unique key for an individual group rule"""
     if isinstance(rule, dict):
@@ -203,10 +211,10 @@ def make_rule_key(prefix, rule, group_id, cidr_ip):
     return key.lower().replace('-none', '-None')
 
 
-def addRulesToLookup(rules, prefix, dict):
+def addRulesToLookup(rules, prefix, rules_dict):
     for rule in rules:
         for grant in rule.grants:
-            dict[make_rule_key(prefix, rule, grant.group_id, grant.cidr_ip)] = (rule, grant)
+            rules_dict[make_rule_key(prefix, rule, grant.group_id, grant.cidr_ip)] = (rule, grant)
 
 
 def validate_rule(module, rule):
@@ -382,8 +390,8 @@ def main():
     name = module.params['name']
     description = module.params['description']
     vpc_id = module.params['vpc_id']
-    rules = rules_expand_sources(rules_expand_ports(module.params['rules']))
-    rules_egress = rules_expand_sources(rules_expand_ports(module.params['rules_egress']))
+    rules = deduplicate_rules_args(rules_expand_sources(rules_expand_ports(module.params['rules'])))
+    rules_egress = deduplicate_rules_args(rules_expand_sources(rules_expand_ports(module.params['rules_egress'])))
     state = module.params.get('state')
     purge_rules = module.params['purge_rules']
     purge_rules_egress = module.params['purge_rules_egress']
@@ -486,10 +494,7 @@ def main():
                 # If rule already exists, don't later delete it
                 for thisip in ip:
                     ruleId = make_rule_key('in', rule, group_id, thisip)
-                    if ruleId in groupRules:
-                        del groupRules[ruleId]
-                    # Otherwise, add new rule
-                    else:
+                    if ruleId not in groupRules:
                         grantGroup = None
                         if group_id:
                             grantGroup = groups[group_id]
@@ -497,6 +502,8 @@ def main():
                         if not module.check_mode:
                             group.authorize(rule['proto'], rule['from_port'], rule['to_port'], thisip, grantGroup)
                         changed = True
+                    else:
+                        del groupRules[ruleId]
 
         # Finally, remove anything left in the groupRules -- these will be defunct rules
         if purge_rules:


Test patch: