Repository: ansible/ansible

Issue:
ec2_group doesn't allow for target of prefix list.
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Feature Idea

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.3.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.10 (default, Jul 15 2017, 17:16:57) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->
Nada

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
Mac 10.13 (High Sierra)

##### SUMMARY
<!--- Explain the problem briefly -->

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
    - name: Some SG name
      local_action:
        module: ec2_group
        profile: xxxxx
        name: xxxxxxx
        description: xxxxxx
        vpc_id: vpc-xxxxxxxx
        region: some region
        state: present
        rules: []
        rules_egress:
          - proto: tcp
            from_port: 80
            to_port: 80
            prefix_list: pl-xxxxxx
          - proto: tcp
            from_port: 443
            to_port: 443
            prefix_list: pl-xxxxxx
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Feature request would like to apply prefix list to a target within an SG.
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```

```



Discussion:

Resolving patch:
diff --git a/lib/ansible/module_utils/aws/iam.py b/lib/ansible/module_utils/aws/iam.py
new file mode 100644
--- /dev/null
+++ b/lib/ansible/module_utils/aws/iam.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2017 Ansible Project
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+import traceback
+
+try:
+    from botocore.exceptions import ClientError, NoCredentialsError
+except ImportError:
+    pass  # caught by HAS_BOTO3
+
+from ansible.module_utils._text import to_native
+
+
+def get_aws_account_id(module):
+    """ Given AnsibleAWSModule instance, get the active AWS account ID
+
+    get_account_id tries too find out the account that we are working
+    on.  It's not guaranteed that this will be easy so we try in
+    several different ways.  Giving either IAM or STS privilages to
+    the account should be enough to permit this.
+    """
+    account_id = None
+    try:
+        sts_client = module.client('sts')
+        account_id = sts_client.get_caller_identity().get('Account')
+    # non-STS sessions may also get NoCredentialsError from this STS call, so
+    # we must catch that too and try the IAM version
+    except (ClientError, NoCredentialsError):
+        try:
+            iam_client = module.client('iam')
+            account_id = iam_client.get_user()['User']['Arn'].split(':')[4]
+        except ClientError as e:
+            if (e.response['Error']['Code'] == 'AccessDenied'):
+                except_msg = to_native(e)
+                # don't match on `arn:aws` because of China region `arn:aws-cn` and similar
+                account_id = except_msg.search(r"arn:\w+:iam::([0-9]{12,32}):\w+/").group(1)
+            if account_id is None:
+                module.fail_json_aws(e, msg="Could not get AWS account information")
+        except Exception as e:
+            module.fail_json(
+                msg="Failed to get AWS account information, Try allowing sts:GetCallerIdentity or iam:GetUser permissions.",
+                exception=traceback.format_exc()
+            )
+    if not account_id:
+        module.fail_json(msg="Failed while determining AWS account ID. Try allowing sts:GetCallerIdentity or iam:GetUser permissions.")
+    return to_native(account_id)
diff --git a/lib/ansible/module_utils/aws/waiters.py b/lib/ansible/module_utils/aws/waiters.py
--- a/lib/ansible/module_utils/aws/waiters.py
+++ b/lib/ansible/module_utils/aws/waiters.py
@@ -27,6 +27,24 @@
                 },
             ]
         },
+        "SecurityGroupExists": {
+            "delay": 5,
+            "maxAttempts": 40,
+            "operation": "DescribeSecurityGroups",
+            "acceptors": [
+                {
+                    "matcher": "path",
+                    "expected": True,
+                    "argument": "length(SecurityGroups[]) > `0`",
+                    "state": "success"
+                },
+                {
+                    "matcher": "error",
+                    "expected": "InvalidGroup.NotFound",
+                    "state": "retry"
+                },
+            ]
+        },
         "SubnetExists": {
             "delay": 5,
             "maxAttempts": 40,
@@ -161,6 +179,12 @@ def waf_model(name):
         core_waiter.NormalizedOperationMethod(
             ec2.describe_route_tables
         )),
+    ('EC2', 'security_group_exists'): lambda ec2: core_waiter.Waiter(
+        'security_group_exists',
+        ec2_model('SecurityGroupExists'),
+        core_waiter.NormalizedOperationMethod(
+            ec2.describe_security_groups
+        )),
     ('EC2', 'subnet_exists'): lambda ec2: core_waiter.Waiter(
         'subnet_exists',
         ec2_model('SubnetExists'),
diff --git a/lib/ansible/modules/cloud/amazon/ec2_group.py b/lib/ansible/modules/cloud/amazon/ec2_group.py
--- a/lib/ansible/modules/cloud/amazon/ec2_group.py
+++ b/lib/ansible/modules/cloud/amazon/ec2_group.py
@@ -147,6 +147,8 @@
       - proto: tcp
         from_port: 443
         to_port: 443
+        # this should only be needed for EC2 Classic security group rules
+        # because in a VPC an ELB will use a user-account security group
         group_id: amazon-elb/sg-87654321/amazon-elb-sg
       - proto: tcp
         from_port: 3306
@@ -288,21 +290,135 @@
 
 import json
 import re
-from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils.ec2 import boto3_conn
-from ansible.module_utils.ec2 import get_aws_connection_info
-from ansible.module_utils.ec2 import ec2_argument_spec
-from ansible.module_utils.ec2 import camel_dict_to_snake_dict
-from ansible.module_utils.ec2 import HAS_BOTO3
-from ansible.module_utils.ec2 import boto3_tag_list_to_ansible_dict, ansible_dict_to_boto3_tag_list, compare_aws_tags
-from ansible.module_utils.ec2 import AWSRetry
+from time import sleep
+from collections import namedtuple
+from ansible.module_utils.aws.core import AnsibleAWSModule
+from ansible.module_utils.aws.iam import get_aws_account_id
+from ansible.module_utils.aws.waiters import get_waiter
+from ansible.module_utils.ec2 import AWSRetry, camel_dict_to_snake_dict, compare_aws_tags
+from ansible.module_utils.ec2 import ansible_dict_to_boto3_filter_list, boto3_tag_list_to_ansible_dict, ansible_dict_to_boto3_tag_list
 from ansible.module_utils.network.common.utils import to_ipv6_network, to_subnet
-import traceback
+from ansible.module_utils._text import to_text
+from ansible.module_utils.six import string_types
 
 try:
-    import botocore
+    from botocore.exceptions import BotoCoreError, ClientError
 except ImportError:
-    pass  # caught by imported HAS_BOTO3
+    pass  # caught by AnsibleAWSModule
+
+
+Rule = namedtuple('Rule', ['port_range', 'protocol', 'target', 'target_type', 'description'])
+valid_targets = set(['ipv4', 'ipv6', 'group', 'ip_prefix'])
+current_account_id = None
+
+
+def rule_cmp(a, b):
+    """Compare rules without descriptions"""
+    for prop in ['port_range', 'protocol', 'target', 'target_type']:
+        if getattr(a, prop) != getattr(b, prop):
+            return False
+    return True
+
+
+def rules_to_permissions(rules):
+    return [to_permission(rule) for rule in rules]
+
+
+def to_permission(rule):
+    # take a Rule, output the serialized grant
+    perm = {
+        'IpProtocol': rule.protocol,
+    }
+    perm['FromPort'], perm['ToPort'] = rule.port_range
+    if rule.target_type == 'ipv4':
+        perm['IpRanges'] = [{
+            'CidrIp': rule.target,
+        }]
+        if rule.description:
+            perm['IpRanges'][0]['Description'] = rule.description
+    elif rule.target_type == 'ipv6':
+        perm['Ipv6Ranges'] = [{
+            'CidrIpv6': rule.target,
+        }]
+        if rule.description:
+            perm['Ipv6Ranges'][0]['Description'] = rule.description
+    elif rule.target_type == 'group':
+        if isinstance(rule.target, tuple):
+            pair = {}
+            if rule.target[0]:
+                pair['UserId'] = rule.target[0]
+            # groupid/groupname are mutually exclusive
+            if rule.target[1] and not rule.target[2]:
+                pair['GroupId'] = rule.target[1]
+            if rule.target[2]:
+                pair['GroupName'] = rule.target[2]
+            perm['UserIdGroupPairs'] = [pair]
+        else:
+            perm['UserIdGroupPairs'] = [{
+                'GroupId': rule.target
+            }]
+        if rule.description:
+            perm['UserIdGroupPairs'][0]['Description'] = rule.description
+    elif rule.target_type == 'ip_prefix':
+        perm['PrefixListIds'] = [{
+            'PrefixListId': rule.target,
+        }]
+        if rule.description:
+            perm['PrefixListIds'][0]['Description'] = rule.description
+    elif rule.target_type not in valid_targets:
+        raise ValueError('Invalid target type for rule {0}'.format(rule))
+    return fix_port_and_protocol(perm)
+
+
+def rule_from_group_permission(perm):
+    def ports_from_permission(p):
+        if 'FromPort' not in p and 'ToPort' not in p:
+            return (None, None)
+        return (int(perm['FromPort']), int(perm['ToPort']))
+
+    # outputs a rule tuple
+    for target_key, target_subkey, target_type in [
+        ('IpRanges', 'CidrIp', 'ipv4'),
+        ('Ipv6Ranges', 'CidrIpv6', 'ipv6'),
+        ('PrefixListIds', 'PrefixListId', 'ip_prefix'),
+    ]:
+        if target_key not in perm:
+            continue
+        for r in perm[target_key]:
+            # there may be several IP ranges here, which is ok
+            yield Rule(
+                ports_from_permission(perm),
+                perm['IpProtocol'],
+                r[target_subkey],
+                target_type,
+                r.get('Description')
+            )
+    if 'UserIdGroupPairs' in perm and perm['UserIdGroupPairs']:
+        for pair in perm['UserIdGroupPairs']:
+            target = pair['GroupId']
+            if pair.get('UserId') and pair['UserId'] != current_account_id:
+                target = (
+                    pair.get('UserId', None),
+                    pair.get('GroupId', None),
+                    pair.get('GroupName', None),
+                )
+            if pair.get('UserId', '').startswith('amazon-'):
+                # amazon-elb and amazon-prefix rules don't need
+                # group-id specified, so remove it when querying
+                # from permission
+                target = (
+                    target[0],
+                    None,
+                    target[2],
+                )
+
+            yield Rule(
+                ports_from_permission(perm),
+                perm['IpProtocol'],
+                target,
+                'group',
+                pair.get('Description')
+            )
 
 
 @AWSRetry.backoff(tries=5, delay=5, backoff=2.0)
@@ -310,6 +426,14 @@ def get_security_groups_with_backoff(connection, **kwargs):
     return connection.describe_security_groups(**kwargs)
 
 
+@AWSRetry.backoff(tries=5, delay=5, backoff=2.0)
+def sg_exists_with_backoff(connection, **kwargs):
+    try:
+        return connection.describe_security_groups(**kwargs)
+    except connection.exceptions.from_code('InvalidGroup.NotFound') as e:
+        return {'SecurityGroups': []}
+
+
 def deduplicate_rules_args(rules):
     """Returns unique rules"""
     if rules is None:
@@ -317,37 +441,15 @@ def deduplicate_rules_args(rules):
     return list(dict(zip((json.dumps(r, sort_keys=True) for r in rules), rules)).values())
 
 
-def make_rule_key(prefix, rule, group_id, cidr_ip):
-    if 'proto' in rule:
-        proto, from_port, to_port = [rule.get(x, None) for x in ('proto', 'from_port', 'to_port')]
-    elif 'IpProtocol' in rule:
-        proto, from_port, to_port = [rule.get(x, None) for x in ('IpProtocol', 'FromPort', 'ToPort')]
-    if proto not in ['icmp', 'tcp', 'udp'] and from_port == -1 and to_port == -1:
-        from_port = 'none'
-        to_port = 'none'
-    key = "%s-%s-%s-%s-%s-%s" % (prefix, proto, from_port, to_port, group_id, cidr_ip)
-    return key.lower().replace('-none', '-None')
-
-
-def add_rules_to_lookup(ipPermissions, group_id, prefix, dict):
-    for rule in ipPermissions:
-        for groupGrant in rule.get('UserIdGroupPairs', []):
-            dict[make_rule_key(prefix, rule, group_id, groupGrant.get('GroupId'))] = (rule, groupGrant)
-        for ipv4Grants in rule.get('IpRanges', []):
-            dict[make_rule_key(prefix, rule, group_id, ipv4Grants.get('CidrIp'))] = (rule, ipv4Grants)
-        for ipv6Grants in rule.get('Ipv6Ranges', []):
-            dict[make_rule_key(prefix, rule, group_id, ipv6Grants.get('CidrIpv6'))] = (rule, ipv6Grants)
-
-
 def validate_rule(module, rule):
-    VALID_PARAMS = ('cidr_ip', 'cidr_ipv6',
+    VALID_PARAMS = ('cidr_ip', 'cidr_ipv6', 'ip_prefix',
                     'group_id', 'group_name', 'group_desc',
                     'proto', 'from_port', 'to_port', 'rule_desc')
     if not isinstance(rule, dict):
         module.fail_json(msg='Invalid rule parameter type [%s].' % type(rule))
     for k in rule:
         if k not in VALID_PARAMS:
-            module.fail_json(msg='Invalid rule parameter \'{}\''.format(k))
+            module.fail_json(msg='Invalid rule parameter \'{0}\' for rule: {1}'.format(k, rule))
 
     if 'group_id' in rule and 'cidr_ip' in rule:
         module.fail_json(msg='Specify group_id OR cidr_ip, not both')
@@ -365,7 +467,7 @@ def validate_rule(module, rule):
 
 def get_target_from_rule(module, client, rule, name, group, groups, vpc_id):
     """
-    Returns tuple of (group_id, ip) after validating rule params.
+    Returns tuple of (target_type, target, group_created) after validating rule params.
 
     rule: Dict describing a rule.
     name: Name of the security group being managed.
@@ -375,34 +477,23 @@ def get_target_from_rule(module, client, rule, name, group, groups, vpc_id):
     function validate the rule specification and return either a non-None
     group_id or a non-None ip range.
     """
-
-    FOREIGN_SECURITY_GROUP_REGEX = r'^(\S+)/(sg-\S+)/(\S+)'
+    FOREIGN_SECURITY_GROUP_REGEX = r'^([^/]+)/?(sg-\S+)?/(\S+)'
     group_id = None
     group_name = None
-    ip = None
-    ipv6 = None
     target_group_created = False
 
-    if 'group_id' in rule and 'cidr_ip' in rule:
-        module.fail_json(msg="Specify group_id OR cidr_ip, not both")
-    elif 'group_name' in rule and 'cidr_ip' in rule:
-        module.fail_json(msg="Specify group_name OR cidr_ip, not both")
-    elif 'group_id' in rule and 'cidr_ipv6' in rule:
-        module.fail_json(msg="Specify group_id OR cidr_ipv6, not both")
-    elif 'group_name' in rule and 'cidr_ipv6' in rule:
-        module.fail_json(msg="Specify group_name OR cidr_ipv6, not both")
-    elif 'group_id' in rule and 'group_name' in rule:
-        module.fail_json(msg="Specify group_id OR group_name, not both")
-    elif 'cidr_ip' in rule and 'cidr_ipv6' in rule:
-        module.fail_json(msg="Specify cidr_ip OR cidr_ipv6, not both")
-    elif rule.get('group_id') and re.match(FOREIGN_SECURITY_GROUP_REGEX, rule['group_id']):
+    validate_rule(module, rule)
+    if rule.get('group_id') and re.match(FOREIGN_SECURITY_GROUP_REGEX, rule['group_id']) and current_account_id not in rule['group_id']:
         # this is a foreign Security Group. Since you can't fetch it you must create an instance of it
         owner_id, group_id, group_name = re.match(FOREIGN_SECURITY_GROUP_REGEX, rule['group_id']).groups()
-        group_instance = dict(GroupId=group_id, GroupName=group_name)
+        group_instance = dict(UserId=owner_id, GroupId=group_id, GroupName=group_name)
         groups[group_id] = group_instance
         groups[group_name] = group_instance
+        if group_id and group_name:
+            group_id = None
+        return 'group', (owner_id, group_id, group_name), False
     elif 'group_id' in rule:
-        group_id = rule['group_id']
+        return 'group', rule['group_id'], False
     elif 'group_name' in rule:
         group_name = rule['group_name']
         if group_name == name:
@@ -416,40 +507,72 @@ def get_target_from_rule(module, client, rule, name, group, groups, vpc_id):
             # both are EC2 classic, this is ok
             group_id = groups[group_name]['GroupId']
         else:
+            auto_group = None
+            filters = {'group-name': group_name}
+            if vpc_id:
+                filters['vpc-id'] = vpc_id
             # if we got here, either the target group does not exist, or there
             # is a mix of EC2 classic + VPC groups. Mixing of EC2 classic + VPC
             # is bad, so we have to create a new SG because no compatible group
             # exists
             if not rule.get('group_desc', '').strip():
-                module.fail_json(msg="group %s will be automatically created by rule %s and "
-                                     "no description was provided" % (group_name, rule))
-            if not module.check_mode:
+                # retry describing the group once
+                try:
+                    auto_group = get_security_groups_with_backoff(client, Filters=ansible_dict_to_boto3_filter_list(filters)).get('SecurityGroups', [])[0]
+                except (client.exceptions.from_code('InvalidGroup.NotFound'), IndexError) as e:
+                    module.fail_json(msg="group %s will be automatically created by rule %s but "
+                                         "no description was provided" % (group_name, rule))
+                except ClientError as e:
+                    module.fail_json_aws(e)
+            elif not module.check_mode:
                 params = dict(GroupName=group_name, Description=rule['group_desc'])
                 if vpc_id:
                     params['VpcId'] = vpc_id
-                auto_group = client.create_security_group(**params)
+                try:
+                    auto_group = client.create_security_group(**params)
+                    get_waiter(
+                        client, 'security_group_exists',
+                    ).wait(
+                        GroupIds=[auto_group['GroupId']],
+                    )
+                except client.exceptions.from_code('InvalidGroup.Duplicate') as e:
+                    # The group exists, but didn't show up in any of our describe-security-groups calls
+                    # Try searching on a filter for the name, and allow a retry window for AWS to update
+                    # the model on their end.
+                    try:
+                        auto_group = get_security_groups_with_backoff(client, Filters=ansible_dict_to_boto3_filter_list(filters)).get('SecurityGroups', [])[0]
+                    except IndexError as e:
+                        module.fail_json(msg="Could not create or use existing group '{0}' in rule. Make sure the group exists".format(group_name))
+                    except ClientError as e:
+                        module.fail_json_aws(
+                            e,
+                            msg="Could not create or use existing group '{0}' in rule. Make sure the group exists".format(group_name))
+            if auto_group is not None:
                 group_id = auto_group['GroupId']
                 groups[group_id] = auto_group
                 groups[group_name] = auto_group
             target_group_created = True
+        return 'group', group_id, target_group_created
     elif 'cidr_ip' in rule:
-        ip = rule['cidr_ip']
+        return 'ipv4', validate_ip(module, rule['cidr_ip']), False
     elif 'cidr_ipv6' in rule:
-        ipv6 = rule['cidr_ipv6']
+        return 'ipv6', validate_ip(module, rule['cidr_ipv6']), False
+    elif 'ip_prefix' in rule:
+        return 'ip_prefix', rule['ip_prefix'], False
 
-    return group_id, ip, ipv6, target_group_created
+    module.fail_json(msg="Could not match target for rule {0}".format(rule), failed_rule=rule)
 
 
 def ports_expand(ports):
     # takes a list of ports and returns a list of (port_from, port_to)
     ports_expanded = []
     for port in ports:
-        if not isinstance(port, str):
+        if not isinstance(port, string_types):
             ports_expanded.append((port,) * 2)
         elif '-' in port:
-            ports_expanded.append(tuple(p.strip() for p in port.split('-', 1)))
+            ports_expanded.append(tuple(int(p.strip()) for p in port.split('-', 1)))
         else:
-            ports_expanded.append((port.strip(),) * 2)
+            ports_expanded.append((int(port.strip()),) * 2)
 
     return ports_expanded
 
@@ -457,6 +580,10 @@ def ports_expand(ports):
 def rule_expand_ports(rule):
     # takes a rule dict and returns a list of expanded rule dicts
     if 'ports' not in rule:
+        if isinstance(rule.get('from_port'), string_types):
+            rule['from_port'] = int(rule.get('from_port'))
+        if isinstance(rule.get('to_port'), string_types):
+            rule['to_port'] = int(rule.get('to_port'))
         return [rule]
 
     ports = rule['ports'] if isinstance(rule['ports'], list) else [rule['ports']]
@@ -465,7 +592,7 @@ def rule_expand_ports(rule):
     for from_to in ports_expand(ports):
         temp_rule = rule.copy()
         del temp_rule['ports']
-        temp_rule['from_port'], temp_rule['to_port'] = from_to
+        temp_rule['from_port'], temp_rule['to_port'] = sorted(from_to)
         rule_expanded.append(temp_rule)
 
     return rule_expanded
@@ -483,7 +610,7 @@ def rules_expand_ports(rules):
 def rule_expand_source(rule, source_type):
     # takes a rule dict and returns a list of expanded rule dicts for specified source_type
     sources = rule[source_type] if isinstance(rule[source_type], list) else [rule[source_type]]
-    source_types_all = ('cidr_ip', 'cidr_ipv6', 'group_id', 'group_name')
+    source_types_all = ('cidr_ip', 'cidr_ipv6', 'group_id', 'group_name', 'ip_prefix')
 
     rule_expanded = []
     for source in sources:
@@ -498,7 +625,7 @@ def rule_expand_source(rule, source_type):
 
 def rule_expand_sources(rule):
     # takes a rule dict and returns a list of expanded rule discts
-    source_types = (stype for stype in ('cidr_ip', 'cidr_ipv6', 'group_id', 'group_name') if stype in rule)
+    source_types = (stype for stype in ('cidr_ip', 'cidr_ipv6', 'group_id', 'group_name', 'ip_prefix') if stype in rule)
 
     return [r for stype in source_types
             for r in rule_expand_source(rule, stype)]
@@ -514,155 +641,214 @@ def rules_expand_sources(rules):
 
 
 def update_rules_description(module, client, rule_type, group_id, ip_permissions):
+    if module.check_mode:
+        return
     try:
         if rule_type == "in":
-            client.update_security_group_rule_descriptions_ingress(GroupId=group_id, IpPermissions=[ip_permissions])
+            client.update_security_group_rule_descriptions_ingress(GroupId=group_id, IpPermissions=ip_permissions)
         if rule_type == "out":
-            client.update_security_group_rule_descriptions_egress(GroupId=group_id, IpPermissions=[ip_permissions])
-    except botocore.exceptions.ClientError as e:
-        module.fail_json(
-            msg="Unable to update rule description for group %s: %s" %
-                (group_id, e),
-            exceptin=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-
-
-def authorize_ip(type, changed, client, group, groupRules,
-                 ip, ip_permission, module, rule, ethertype):
-    # If rule already exists, don't later delete it
-    for this_ip in ip:
-
-        split_addr = this_ip.split('/')
-        if len(split_addr) == 2:
-            # this_ip is a IPv4 or IPv6 CIDR that may or may not have host bits set
-            # Get the network bits.
-            try:
-                thisip = to_subnet(split_addr[0], split_addr[1])
-            except ValueError:
-                thisip = to_ipv6_network(split_addr[0]) + "/" + split_addr[1]
-            if thisip != this_ip:
-                module.warn("One of your CIDR addresses ({0}) has host bits set. To get rid of this warning, "
-                            "check the network mask and make sure that only network bits are set: {1}.".format(this_ip, thisip))
-        else:
-            thisip = this_ip
-
-        rule_id = make_rule_key(type, rule, group['GroupId'], thisip)
-        if rule_id in groupRules:
-
-            # update the rule description
-            if 'rule_desc' in rule:
-                desired_rule_desc = rule.get('rule_desc') or ''
-                current_rule = groupRules[rule_id][0].get('IpRanges') or groupRules[rule_id][0].get('Ipv6Ranges')
-                if desired_rule_desc != current_rule[0].get('Description', ''):
-                    if not module.check_mode:
-                        ip_permission = serialize_ip_grant(rule, thisip, ethertype)
-                        update_rules_description(module, client, type, group['GroupId'], ip_permission)
-                    changed = True
-
-            # remove the rule from groupRules to avoid purging it later
-            del groupRules[rule_id]
-        else:
-            if not module.check_mode:
-                ip_permission = serialize_ip_grant(rule, thisip, ethertype)
-                if ip_permission:
-                    try:
-                        if type == "in":
-                            client.authorize_security_group_ingress(GroupId=group['GroupId'],
-                                                                    IpPermissions=[ip_permission])
-                        elif type == "out":
-                            client.authorize_security_group_egress(GroupId=group['GroupId'],
-                                                                   IpPermissions=[ip_permission])
-                    except botocore.exceptions.ClientError as e:
-                        module.fail_json(msg="Unable to authorize %s for ip %s security group '%s' - %s" %
-                                             (type, thisip, group['GroupName'], e),
-                                         exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-            changed = True
-    return changed, ip_permission
-
-
-def serialize_group_grant(group_id, rule):
-    permission = {'IpProtocol': rule['proto'],
-                  'FromPort': rule['from_port'],
-                  'ToPort': rule['to_port'],
-                  'UserIdGroupPairs': [{'GroupId': group_id}]}
-
-    if 'rule_desc' in rule:
-        permission['UserIdGroupPairs'][0]['Description'] = rule.get('rule_desc') or ''
-
-    return fix_port_and_protocol(permission)
-
-
-def serialize_revoke(grant, rule):
-    permission = dict()
-    fromPort = rule['FromPort'] if 'FromPort' in rule else None
-    toPort = rule['ToPort'] if 'ToPort' in rule else None
-    if 'GroupId' in grant:
-        permission = {'IpProtocol': rule['IpProtocol'],
-                      'FromPort': fromPort,
-                      'ToPort': toPort,
-                      'UserIdGroupPairs': [{'GroupId': grant['GroupId']}]
-                      }
-    elif 'CidrIp' in grant:
-        permission = {'IpProtocol': rule['IpProtocol'],
-                      'FromPort': fromPort,
-                      'ToPort': toPort,
-                      'IpRanges': [grant]
-                      }
-    elif 'CidrIpv6' in grant:
-        permission = {'IpProtocol': rule['IpProtocol'],
-                      'FromPort': fromPort,
-                      'ToPort': toPort,
-                      'Ipv6Ranges': [grant]
-                      }
-    return fix_port_and_protocol(permission)
-
-
-def serialize_ip_grant(rule, thisip, ethertype):
-    permission = {'IpProtocol': rule['proto'],
-                  'FromPort': rule['from_port'],
-                  'ToPort': rule['to_port']}
-    if ethertype == "ipv4":
-        permission['IpRanges'] = [{'CidrIp': thisip}]
-        if 'rule_desc' in rule:
-            permission['IpRanges'][0]['Description'] = rule.get('rule_desc') or ''
-    elif ethertype == "ipv6":
-        permission['Ipv6Ranges'] = [{'CidrIpv6': thisip}]
-        if 'rule_desc' in rule:
-            permission['Ipv6Ranges'][0]['Description'] = rule.get('rule_desc') or ''
-
-    return fix_port_and_protocol(permission)
+            client.update_security_group_rule_descriptions_egress(GroupId=group_id, IpPermissions=ip_permissions)
+    except (ClientError, BotoCoreError) as e:
+        module.fail_json_aws(e, msg="Unable to update rule description for group %s" % group_id)
 
 
 def fix_port_and_protocol(permission):
-    for key in ['FromPort', 'ToPort']:
+    for key in ('FromPort', 'ToPort'):
         if key in permission:
             if permission[key] is None:
                 del permission[key]
             else:
                 permission[key] = int(permission[key])
 
-    permission['IpProtocol'] = str(permission['IpProtocol'])
+    permission['IpProtocol'] = to_text(permission['IpProtocol'])
 
     return permission
 
 
-def check_rule_desc_update_for_group_grant(client, module, rule, group, groupRules, rule_id, group_id, rule_type, changed):
-    if 'rule_desc' in rule:
-        current_rule_description = rule.get('rule_desc') or ''
-        if current_rule_description != groupRules[rule_id][0]['UserIdGroupPairs'][0].get('Description', ''):
-            if not module.check_mode:
-                ip_permission = serialize_group_grant(group_id, rule)
-                update_rules_description(module, client, rule_type, group['GroupId'], ip_permission)
-            changed = True
+def remove_old_permissions(client, module, revoke_ingress, revoke_egress, group_id):
+    if revoke_ingress:
+        revoke(client, module, revoke_ingress, group_id, 'in')
+    if revoke_egress:
+        revoke(client, module, revoke_egress, group_id, 'out')
+    return bool(revoke_ingress or revoke_egress)
+
+
+def revoke(client, module, ip_permissions, group_id, rule_type):
+    if not module.check_mode:
+        try:
+            if rule_type == 'in':
+                client.revoke_security_group_ingress(GroupId=group_id, IpPermissions=ip_permissions)
+            elif rule_type == 'out':
+                client.revoke_security_group_egress(GroupId=group_id, IpPermissions=ip_permissions)
+        except (BotoCoreError, ClientError) as e:
+            rules = 'ingress rules' if rule_type == 'in' else 'egress rules'
+            module.fail_json_aws(e, "Unable to revoke {0}: {1}".format(rules, ip_permissions))
+
+
+def add_new_permissions(client, module, new_ingress, new_egress, group_id):
+    if new_ingress:
+        authorize(client, module, new_ingress, group_id, 'in')
+    if new_egress:
+        authorize(client, module, new_egress, group_id, 'out')
+    return bool(new_ingress or new_egress)
+
+
+def authorize(client, module, ip_permissions, group_id, rule_type):
+    if not module.check_mode:
+        try:
+            if rule_type == 'in':
+                client.authorize_security_group_ingress(GroupId=group_id, IpPermissions=ip_permissions)
+            elif rule_type == 'out':
+                client.authorize_security_group_egress(GroupId=group_id, IpPermissions=ip_permissions)
+        except (BotoCoreError, ClientError) as e:
+            rules = 'ingress rules' if rule_type == 'in' else 'egress rules'
+            module.fail_json_aws(e, "Unable to authorize {0}: {1}".format(rules, ip_permissions))
+
+
+def validate_ip(module, cidr_ip):
+    split_addr = cidr_ip.split('/')
+    if len(split_addr) == 2:
+        # this_ip is a IPv4 or IPv6 CIDR that may or may not have host bits set
+        # Get the network bits.
+        try:
+            ip = to_subnet(split_addr[0], split_addr[1])
+        except ValueError:
+            ip = to_ipv6_network(split_addr[0]) + "/" + split_addr[1]
+        if ip != cidr_ip:
+            module.warn("One of your CIDR addresses ({0}) has host bits set. To get rid of this warning, "
+                        "check the network mask and make sure that only network bits are set: {1}.".format(cidr_ip, ip))
+        return ip
+    return cidr_ip
+
+
+def update_tags(client, module, group_id, current_tags, tags, purge_tags):
+    tags_need_modify, tags_to_delete = compare_aws_tags(current_tags, tags, purge_tags)
+
+    if not module.check_mode:
+        if tags_to_delete:
+            try:
+                client.delete_tags(Resources=[group_id], Tags=[{'Key': tag} for tag in tags_to_delete])
+            except (BotoCoreError, ClientError) as e:
+                module.fail_json_aws(e, msg="Unable to delete tags {0}".format(tags_to_delete))
+
+        # Add/update tags
+        if tags_need_modify:
+            try:
+                client.create_tags(Resources=[group_id], Tags=ansible_dict_to_boto3_tag_list(tags_need_modify))
+            except (BotoCoreError, ClientError) as e:
+                module.fail_json(e, msg="Unable to add tags {0}".format(tags_need_modify))
+
+    return bool(tags_need_modify or tags_to_delete)
+
+
+def update_rule_descriptions(module, group_id, present_ingress, named_tuple_ingress_list, present_egress, named_tuple_egress_list):
+    changed = False
+    client = module.client('ec2')
+    ingress_needs_desc_update = []
+    egress_needs_desc_update = []
+
+    for present_rule in present_egress:
+        needs_update = [r for r in named_tuple_egress_list if rule_cmp(r, present_rule) and r.description != present_rule.description]
+        for r in needs_update:
+            named_tuple_egress_list.remove(r)
+        egress_needs_desc_update.extend(needs_update)
+    for present_rule in present_ingress:
+        needs_update = [r for r in named_tuple_ingress_list if rule_cmp(r, present_rule) and r.description != present_rule.description]
+        for r in needs_update:
+            named_tuple_ingress_list.remove(r)
+        ingress_needs_desc_update.extend(needs_update)
+
+    if ingress_needs_desc_update:
+        update_rules_description(module, client, 'in', group_id, rules_to_permissions(ingress_needs_desc_update))
+        changed |= True
+    if egress_needs_desc_update:
+        update_rules_description(module, client, 'out', group_id, rules_to_permissions(egress_needs_desc_update))
+        changed |= True
     return changed
 
 
-def has_rule_description_attr(client):
-    return hasattr(client, "update_security_group_rule_descriptions_egress")
+def create_security_group(client, module, name, description, vpc_id):
+    if not module.check_mode:
+        params = dict(GroupName=name, Description=description)
+        if vpc_id:
+            params['VpcId'] = vpc_id
+        try:
+            group = client.create_security_group(**params)
+        except (BotoCoreError, ClientError) as e:
+            module.fail_json_aws(e, msg="Unable to create security group")
+        # When a group is created, an egress_rule ALLOW ALL
+        # to 0.0.0.0/0 is added automatically but it's not
+        # reflected in the object returned by the AWS API
+        # call. We re-read the group for getting an updated object
+        # amazon sometimes takes a couple seconds to update the security group so wait till it exists
+        while True:
+            sleep(3)
+            group = get_security_groups_with_backoff(client, GroupIds=[group['GroupId']])['SecurityGroups'][0]
+            if group.get('VpcId') and not group.get('IpPermissionsEgress'):
+                pass
+            else:
+                break
+        return group
+    return None
+
+
+def wait_for_rule_propagation(module, group, desired_ingress, desired_egress, purge_ingress, purge_egress):
+    group_id = group['GroupId']
+    tries = 6
+
+    def await_rules(group, desired_rules, purge, rule_key):
+        for i in range(tries):
+            current_rules = set(sum([list(rule_from_group_permission(p)) for p in group[rule_key]], []))
+            if purge and len(current_rules ^ set(desired_rules)) == 0:
+                return group
+            elif current_rules.issuperset(desired_rules) and not purge:
+                return group
+            sleep(10)
+            group = get_security_groups_with_backoff(module.client('ec2'), GroupIds=[group_id])['SecurityGroups'][0]
+        module.warn("Ran out of time waiting for {0} {1}. Current: {2}, Desired: {3}".format(group_id, rule_key, current_rules, desired_rules))
+        return group
+
+    group = get_security_groups_with_backoff(module.client('ec2'), GroupIds=[group_id])['SecurityGroups'][0]
+    if 'VpcId' in group and module.params.get('rules_egress') is not None:
+        group = await_rules(group, desired_egress, purge_egress, 'IpPermissionsEgress')
+    return await_rules(group, desired_ingress, purge_ingress, 'IpPermissions')
+
+
+def group_exists(client, module, vpc_id, group_id, name):
+    params = {'Filters': []}
+    if group_id:
+        params['GroupIds'] = [group_id]
+    if name:
+        # Add name to filters rather than params['GroupNames']
+        # because params['GroupNames'] only checks the default vpc if no vpc is provided
+        params['Filters'].append({'Name': 'group-name', 'Values': [name]})
+    if vpc_id:
+        params['Filters'].append({'Name': 'vpc-id', 'Values': [vpc_id]})
+    # Don't filter by description to maintain backwards compatibility
+
+    try:
+        security_groups = sg_exists_with_backoff(client, **params).get('SecurityGroups', [])
+        all_groups = get_security_groups_with_backoff(client).get('SecurityGroups', [])
+    except (BotoCoreError, ClientError) as e:
+        module.fail_json_aws(e, msg="Error in describe_security_groups")
+
+    if security_groups:
+        groups = dict((group['GroupId'], group) for group in all_groups)
+        groups.update(dict((group['GroupName'], group) for group in all_groups))
+        # maintain backwards compatibility by using the last matching group
+        return security_groups[-1], groups
+    return None, {}
+
+
+def verify_rules_with_descriptions_permitted(client, module, rules, rules_egress):
+    if not hasattr(client, "update_security_group_rule_descriptions_egress"):
+        all_rules = rules if rules else [] + rules_egress if rules_egress else []
+        if any('rule_desc' in rule for rule in all_rules):
+            module.fail_json(msg="Using rule descriptions requires botocore version >= 1.7.2.")
 
 
 def main():
-    argument_spec = ec2_argument_spec()
-    argument_spec.update(dict(
+    argument_spec = dict(
         name=dict(),
         group_id=dict(),
         description=dict(),
@@ -675,17 +861,13 @@ def main():
         tags=dict(required=False, type='dict', aliases=['resource_tags']),
         purge_tags=dict(default=True, required=False, type='bool')
     )
-    )
-    module = AnsibleModule(
+    module = AnsibleAWSModule(
         argument_spec=argument_spec,
         supports_check_mode=True,
         required_one_of=[['name', 'group_id']],
         required_if=[['state', 'present', ['name']]],
     )
 
-    if not HAS_BOTO3:
-        module.fail_json(msg='boto3 required for this module')
-
     name = module.params['name']
     group_id = module.params['group_id']
     description = module.params['description']
@@ -702,52 +884,14 @@ def main():
         module.fail_json(msg='Must provide description when state is present.')
 
     changed = False
-    region, ec2_url, aws_connect_params = get_aws_connection_info(module, boto3=True)
-    client = boto3_conn(module, conn_type='client', resource='ec2',
-                        endpoint=ec2_url, region=region, **aws_connect_params)
-
-    if not has_rule_description_attr(client):
-        all_rules = rules if rules else [] + rules_egress if rules_egress else []
-        if any('rule_desc' in rule for rule in all_rules):
-            module.fail_json(msg="Using rule descriptions requires botocore version >= 1.7.2.")
+    client = module.client('ec2')
 
-    group = None
-    groups = dict()
-    security_groups = []
-    # do get all security groups
-    # find if the group is present
-    try:
-        response = get_security_groups_with_backoff(client)
-        security_groups = response.get('SecurityGroups', [])
-    except botocore.exceptions.NoCredentialsError as e:
-        module.fail_json(msg="Error in describe_security_groups: %s" % "Unable to locate credentials", exception=traceback.format_exc())
-    except botocore.exceptions.ClientError as e:
-        module.fail_json(msg="Error in describe_security_groups: %s" % e, exception=traceback.format_exc(),
-                         **camel_dict_to_snake_dict(e.response))
-
-    for sg in security_groups:
-        groups[sg['GroupId']] = sg
-        groupName = sg['GroupName']
-        if groupName in groups:
-            # Prioritise groups from the current VPC
-            # even if current VPC is EC2-Classic
-            if groups[groupName].get('VpcId') == vpc_id:
-                # Group saved already matches current VPC, change nothing
-                pass
-            elif vpc_id is None and groups[groupName].get('VpcId') is None:
-                # We're in EC2 classic, and the group already saved is as well
-                # No VPC groups can be used alongside EC2 classic groups
-                pass
-            else:
-                # the current SG stored has no direct match, so we can replace it
-                groups[groupName] = sg
-        else:
-            groups[groupName] = sg
+    verify_rules_with_descriptions_permitted(client, module, rules, rules_egress)
+    group, groups = group_exists(client, module, vpc_id, group_id, name)
+    group_created_new = not bool(group)
 
-        if group_id and sg['GroupId'] == group_id:
-            group = sg
-        elif groupName == name and (vpc_id is None or sg.get('VpcId') == vpc_id):
-            group = sg
+    global current_account_id
+    current_account_id = get_aws_account_id(module)
 
     # Ensure requested group is absent
     if state == 'absent':
@@ -756,9 +900,8 @@ def main():
             try:
                 if not module.check_mode:
                     client.delete_security_group(GroupId=group['GroupId'])
-            except botocore.exceptions.ClientError as e:
-                module.fail_json(msg="Unable to delete security group '%s' - %s" % (group, e),
-                                 exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
+            except (BotoCoreError, ClientError) as e:
+                module.fail_json_aws(e, msg="Unable to delete security group '%s'" % group)
             else:
                 group = None
                 changed = True
@@ -773,215 +916,99 @@ def main():
             if group['Description'] != description:
                 module.warn("Group description does not match existing group. Descriptions cannot be changed without deleting "
                             "and re-creating the security group. Try using state=absent to delete, then rerunning this task.")
-
-        # if the group doesn't exist, create it now
         else:
             # no match found, create it
-            if not module.check_mode:
-                params = dict(GroupName=name, Description=description)
-                if vpc_id:
-                    params['VpcId'] = vpc_id
-                group = client.create_security_group(**params)
-                # When a group is created, an egress_rule ALLOW ALL
-                # to 0.0.0.0/0 is added automatically but it's not
-                # reflected in the object returned by the AWS API
-                # call. We re-read the group for getting an updated object
-                # amazon sometimes takes a couple seconds to update the security group so wait till it exists
-                while True:
-                    group = get_security_groups_with_backoff(client, GroupIds=[group['GroupId']])['SecurityGroups'][0]
-                    if group.get('VpcId') and not group.get('IpPermissionsEgress'):
-                        pass
-                    else:
-                        break
-
+            group = create_security_group(client, module, name, description, vpc_id)
             changed = True
 
         if tags is not None and group is not None:
             current_tags = boto3_tag_list_to_ansible_dict(group.get('Tags', []))
-            tags_need_modify, tags_to_delete = compare_aws_tags(current_tags, tags, purge_tags)
-            if tags_to_delete:
-                if not module.check_mode:
-                    try:
-                        client.delete_tags(Resources=[group['GroupId']], Tags=[{'Key': tag} for tag in tags_to_delete])
-                    except botocore.exceptions.ClientError as e:
-                        module.fail_json(msg=e.message, exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-                changed = True
-
-            # Add/update tags
-            if tags_need_modify:
-                if not module.check_mode:
-                    try:
-                        client.create_tags(Resources=[group['GroupId']], Tags=ansible_dict_to_boto3_tag_list(tags_need_modify))
-                    except botocore.exceptions.ClientError as e:
-                        module.fail_json(msg=e.message, exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-                changed = True
+            changed |= update_tags(client, module, group['GroupId'], current_tags, tags, purge_tags)
 
-    else:
-        module.fail_json(msg="Unsupported state requested: %s" % state)
-
-    # create a lookup for all existing rules on the group
-    ip_permission = []
     if group:
-        # Manage ingress rules
-        groupRules = {}
-        add_rules_to_lookup(group['IpPermissions'], group['GroupId'], 'in', groupRules)
-        # Now, go through all provided rules and ensure they are there.
-        if rules is not None:
-            for rule in rules:
-                validate_rule(module, rule)
-                group_id, ip, ipv6, target_group_created = get_target_from_rule(module, client, rule, name,
-                                                                                group, groups, vpc_id)
-                if target_group_created:
-                    changed = True
-
-                if rule['proto'] in ('all', '-1', -1):
-                    rule['proto'] = -1
+        named_tuple_ingress_list = []
+        named_tuple_egress_list = []
+        current_ingress = sum([list(rule_from_group_permission(p)) for p in group['IpPermissions']], [])
+        current_egress = sum([list(rule_from_group_permission(p)) for p in group['IpPermissionsEgress']], [])
+
+        for new_rules, rule_type, named_tuple_rule_list in [(rules, 'in', named_tuple_ingress_list),
+                                                            (rules_egress, 'out', named_tuple_egress_list)]:
+            if new_rules is None:
+                continue
+            for rule in new_rules:
+                target_type, target, target_group_created = get_target_from_rule(
+                    module, client, rule, name, group, groups, vpc_id)
+                changed |= target_group_created
+
+                if rule.get('proto', 'tcp') in ('all', '-1', -1):
+                    rule['proto'] = '-1'
                     rule['from_port'] = None
                     rule['to_port'] = None
 
-                if group_id:
-                    rule_id = make_rule_key('in', rule, group['GroupId'], group_id)
-                    if rule_id in groupRules:
-                        changed = check_rule_desc_update_for_group_grant(client, module, rule, group, groupRules,
-                                                                         rule_id, group_id, rule_type='in', changed=changed)
-                        del groupRules[rule_id]
-                    else:
-                        if not module.check_mode:
-                            ip_permission = serialize_group_grant(group_id, rule)
-                            if ip_permission:
-                                ips = ip_permission
-                                if vpc_id:
-                                    [useridpair.update({'VpcId': vpc_id}) for useridpair in
-                                     ip_permission.get('UserIdGroupPairs', [])]
-                                try:
-                                    client.authorize_security_group_ingress(GroupId=group['GroupId'], IpPermissions=[ips])
-                                except botocore.exceptions.ClientError as e:
-                                    module.fail_json(
-                                        msg="Unable to authorize ingress for group %s security group '%s' - %s" %
-                                            (group_id, group['GroupName'], e),
-                                        exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-                        changed = True
-                elif ip:
-                    # Convert ip to list we can iterate over
-                    if ip and not isinstance(ip, list):
-                        ip = [ip]
-
-                    changed, ip_permission = authorize_ip("in", changed, client, group, groupRules, ip, ip_permission,
-                                                          module, rule, "ipv4")
-                elif ipv6:
-                    # Convert ip to list we can iterate over
-                    if not isinstance(ipv6, list):
-                        ipv6 = [ipv6]
-                    # If rule already exists, don't later delete it
-                    changed, ip_permission = authorize_ip("in", changed, client, group, groupRules, ipv6, ip_permission,
-                                                          module, rule, "ipv6")
-        # Finally, remove anything left in the groupRules -- these will be defunct rules
-        if purge_rules:
-            for (rule, grant) in groupRules.values():
-                ip_permission = serialize_revoke(grant, rule)
-                if not module.check_mode:
-                    try:
-                        client.revoke_security_group_ingress(GroupId=group['GroupId'], IpPermissions=[ip_permission])
-                    except botocore.exceptions.ClientError as e:
-                        module.fail_json(
-                            msg="Unable to revoke ingress for security group '%s' - %s" %
-                                (group['GroupName'], e),
-                            exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-                changed = True
-
-        # Manage egress rules
-        groupRules = {}
-        add_rules_to_lookup(group['IpPermissionsEgress'], group['GroupId'], 'out', groupRules)
-        # Now, go through all provided rules and ensure they are there.
-        if rules_egress is not None:
-            for rule in rules_egress:
-                validate_rule(module, rule)
-                group_id, ip, ipv6, target_group_created = get_target_from_rule(module, client, rule, name,
-                                                                                group, groups, vpc_id)
-                if target_group_created:
-                    changed = True
-
-                if rule['proto'] in ('all', '-1', -1):
-                    rule['proto'] = -1
-                    rule['from_port'] = None
-                    rule['to_port'] = None
-
-                if group_id:
-                    rule_id = make_rule_key('out', rule, group['GroupId'], group_id)
-                    if rule_id in groupRules:
-                        changed = check_rule_desc_update_for_group_grant(client, module, rule, group, groupRules,
-                                                                         rule_id, group_id, rule_type='out', changed=changed)
-                        del groupRules[rule_id]
-                    else:
-                        if not module.check_mode:
-                            ip_permission = serialize_group_grant(group_id, rule)
-                            if ip_permission:
-                                ips = ip_permission
-                                if vpc_id:
-                                    [useridpair.update({'VpcId': vpc_id}) for useridpair in
-                                     ip_permission.get('UserIdGroupPairs', [])]
-                                try:
-                                    client.authorize_security_group_egress(GroupId=group['GroupId'], IpPermissions=[ips])
-                                except botocore.exceptions.ClientError as e:
-                                    module.fail_json(
-                                        msg="Unable to authorize egress for group %s security group '%s' - %s" %
-                                            (group_id, group['GroupName'], e),
-                                        exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-                        changed = True
-                elif ip:
-                    # Convert ip to list we can iterate over
-                    if not isinstance(ip, list):
-                        ip = [ip]
-                    changed, ip_permission = authorize_ip("out", changed, client, group, groupRules, ip,
-                                                          ip_permission, module, rule, "ipv4")
-                elif ipv6:
-                    # Convert ip to list we can iterate over
-                    if not isinstance(ipv6, list):
-                        ipv6 = [ipv6]
-                    # If rule already exists, don't later delete it
-                    changed, ip_permission = authorize_ip("out", changed, client, group, groupRules, ipv6,
-                                                          ip_permission, module, rule, "ipv6")
-        elif 'VpcId' in group:
+                named_tuple_rule_list.append(
+                    Rule(
+                        port_range=(rule['from_port'], rule['to_port']),
+                        protocol=rule.get('proto', 'tcp'),
+                        target=target, target_type=target_type,
+                        description=rule.get('rule_desc'),
+                    )
+                )
+
+        # List comprehensions for rules to add, rules to modify, and rule ids to determine purging
+        new_ingress_permissions = [to_permission(r) for r in (set(named_tuple_ingress_list) - set(current_ingress))]
+        new_egress_permissions = [to_permission(r) for r in (set(named_tuple_egress_list) - set(current_egress))]
+        present_ingress = list(set(named_tuple_ingress_list).union(set(current_ingress)))
+        present_egress = list(set(named_tuple_egress_list).union(set(current_egress)))
+
+        if module.params.get('rules_egress') is None and 'VpcId' in group:
             # when no egress rules are specified and we're in a VPC,
             # we add in a default allow all out rule, which was the
             # default behavior before egress rules were added
-            default_egress_rule = 'out--1-None-None-' + group['GroupId'] + '-0.0.0.0/0'
-            if default_egress_rule not in groupRules:
-                if not module.check_mode:
-                    ip_permission = [{'IpProtocol': '-1',
-                                      'IpRanges': [{'CidrIp': '0.0.0.0/0'}]
-                                      }
-                                     ]
-                    try:
-                        client.authorize_security_group_egress(GroupId=group['GroupId'], IpPermissions=ip_permission)
-                    except botocore.exceptions.ClientError as e:
-                        module.fail_json(msg="Unable to authorize egress for ip %s security group '%s' - %s" %
-                                             ('0.0.0.0/0',
-                                              group['GroupName'],
-                                              e),
-                                         exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-                changed = True
-            else:
-                # make sure the default egress rule is not removed
-                del groupRules[default_egress_rule]
-
-        # Finally, remove anything left in the groupRules -- these will be defunct rules
-        if purge_rules_egress and 'VpcId' in group:
-            for (rule, grant) in groupRules.values():
-                # we shouldn't be revoking 0.0.0.0 egress
-                if grant != '0.0.0.0/0':
-                    ip_permission = serialize_revoke(grant, rule)
-                    if not module.check_mode:
-                        try:
-                            client.revoke_security_group_egress(GroupId=group['GroupId'], IpPermissions=[ip_permission])
-                        except botocore.exceptions.ClientError as e:
-                            module.fail_json(msg="Unable to revoke egress for ip %s security group '%s' - %s" %
-                                                 (grant, group['GroupName'], e),
-                                             exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
-                    changed = True
+            rule = Rule((None, None), '-1', '0.0.0.0/0', 'ipv4', None)
+            if rule in current_egress:
+                named_tuple_egress_list.append(rule)
+            if rule not in current_egress:
+                current_egress.append(rule)
 
-    if group:
-        security_group = get_security_groups_with_backoff(client, GroupIds=[group['GroupId']])['SecurityGroups'][0]
+        # List comprehensions for rules to add, rules to modify, and rule ids to determine purging
+        present_ingress = list(set(named_tuple_ingress_list).union(set(current_ingress)))
+        present_egress = list(set(named_tuple_egress_list).union(set(current_egress)))
+
+        if purge_rules:
+            revoke_ingress = [to_permission(r) for r in set(present_ingress) - set(named_tuple_ingress_list)]
+        else:
+            revoke_ingress = []
+        if purge_rules_egress and module.params.get('rules_egress') is not None:
+            if module.params.get('rules_egress') is []:
+                revoke_egress = [
+                    to_permission(r) for r in set(present_egress) - set(named_tuple_egress_list)
+                    if r != Rule((None, None), '-1', '0.0.0.0/0', 'ipv4', None)
+                ]
+            else:
+                revoke_egress = [to_permission(r) for r in set(present_egress) - set(named_tuple_egress_list)]
+        else:
+            revoke_egress = []
+
+        changed |= update_rule_descriptions(module, group['GroupId'], present_ingress, named_tuple_ingress_list, present_egress, named_tuple_egress_list)
+
+        # Revoke old rules
+        changed |= remove_old_permissions(client, module, revoke_ingress, revoke_egress, group['GroupId'])
+        rule_msg = 'Revoking {0}, and egress {1}'.format(revoke_ingress, revoke_egress)
+
+        new_ingress_permissions = [to_permission(r) for r in (set(named_tuple_ingress_list) - set(current_ingress))]
+        new_ingress_permissions = rules_to_permissions(set(named_tuple_ingress_list) - set(current_ingress))
+        new_egress_permissions = rules_to_permissions(set(named_tuple_egress_list) - set(current_egress))
+        # Authorize new rules
+        changed |= add_new_permissions(client, module, new_ingress_permissions, new_egress_permissions, group['GroupId'])
+
+        if group_created_new and module.params.get('rules') is None and module.params.get('rules_egress') is None:
+            # A new group with no rules provided is already being awaited.
+            # When it is created we wait for the default egress rule to be added by AWS
+            security_group = get_security_groups_with_backoff(client, GroupIds=[group['GroupId']])['SecurityGroups'][0]
+        elif changed and not module.check_mode:
+            security_group = wait_for_rule_propagation(module, group, named_tuple_ingress_list, named_tuple_egress_list, purge_rules, purge_rules_egress)
+        else:
+            security_group = get_security_groups_with_backoff(client, GroupIds=[group['GroupId']])['SecurityGroups'][0]
         security_group = camel_dict_to_snake_dict(security_group)
         security_group['tags'] = boto3_tag_list_to_ansible_dict(security_group.get('tags', []),
                                                                 tag_name_key_name='key', tag_value_key_name='value')


Test patch:
diff --git a/test/integration/targets/ec2_group/tasks/credential_tests.yml b/test/integration/targets/ec2_group/tasks/credential_tests.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/ec2_group/tasks/credential_tests.yml
@@ -0,0 +1,161 @@
+---
+# A Note about ec2 environment variable name preference:
+#  - EC2_URL -> AWS_URL
+#  - EC2_ACCESS_KEY -> AWS_ACCESS_KEY_ID -> AWS_ACCESS_KEY
+#  - EC2_SECRET_KEY -> AWS_SECRET_ACCESS_KEY -> AWX_SECRET_KEY
+#  - EC2_REGION -> AWS_REGION
+#
+
+# - include: ../../setup_ec2/tasks/common.yml module_name: ec2_group
+
+- block:
+    # ============================================================
+    - name: test failure with no parameters
+      ec2_group:
+      register: result
+      ignore_errors: true
+
+    - name: assert failure with no parameters
+      assert:
+        that:
+           - 'result.failed'
+           - 'result.msg == "one of the following is required: name, group_id"'
+
+    # ============================================================
+    - name: test failure with only name
+      ec2_group:
+        name: '{{ec2_group_name}}'
+      register: result
+      ignore_errors: true
+
+    - name: assert failure with only name
+      assert:
+        that:
+           - 'result.failed'
+           - 'result.msg == "Must provide description when state is present."'
+
+    # ============================================================
+    - name: test failure with only description
+      ec2_group:
+        description: '{{ec2_group_description}}'
+      register: result
+      ignore_errors: true
+
+    - name: assert failure with only description
+      assert:
+        that:
+           - 'result.failed'
+           - 'result.msg == "one of the following is required: name, group_id"'
+
+    # ============================================================
+    - name: test failure with empty description (AWS API requires non-empty string desc)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: ''
+        region: '{{ec2_region}}'
+      register: result
+      ignore_errors: true
+
+    - name: assert failure with empty description
+      assert:
+        that:
+           - 'result.failed'
+           - 'result.msg == "Must provide description when state is present."'
+
+    # ============================================================
+    - name: test valid region parameter
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        region: '{{ec2_region}}'
+      register: result
+      ignore_errors: true
+
+    - name: assert valid region parameter
+      assert:
+        that:
+           - 'result.failed'
+           - '"Unable to locate credentials" in result.msg'
+
+    # ============================================================
+    - name: test environment variable EC2_REGION
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+      environment:
+        EC2_REGION: '{{ec2_region}}'
+      register: result
+      ignore_errors: true
+
+    - name: assert environment variable EC2_REGION
+      assert:
+        that:
+           - 'result.failed'
+           - '"Unable to locate credentials" in result.msg'
+
+    # ============================================================
+    - name: test invalid ec2_url parameter
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+      environment:
+        EC2_URL: bogus.example.com
+      register: result
+      ignore_errors: true
+
+    - name: assert invalid ec2_url parameter
+      assert:
+        that:
+           - 'result.failed'
+           - 'result.msg.startswith("The ec2_group module requires a region")'
+
+    # ============================================================
+    - name: test valid ec2_url parameter
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+      environment:
+        EC2_URL: '{{ec2_url}}'
+      register: result
+      ignore_errors: true
+
+    - name: assert valid ec2_url parameter
+      assert:
+        that:
+           - 'result.failed'
+           - 'result.msg.startswith("The ec2_group module requires a region")'
+
+    # ============================================================
+    - name: test credentials from environment
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+      environment:
+        EC2_REGION: '{{ec2_region}}'
+        EC2_ACCESS_KEY: bogus_access_key
+        EC2_SECRET_KEY: bogus_secret_key
+      register: result
+      ignore_errors: true
+
+    - name: assert ec2_group with valid ec2_url
+      assert:
+        that:
+           - 'result.failed'
+           - '"validate the provided access credentials" in result.msg'
+
+    # ============================================================
+    - name: test credential parameters
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        ec2_region: '{{ec2_region}}'
+        ec2_access_key: 'bogus_access_key'
+        ec2_secret_key: 'bogus_secret_key'
+      register: result
+      ignore_errors: true
+
+    - name: assert credential parameters
+      assert:
+        that:
+           - 'result.failed'
+           - '"validate the provided access credentials" in result.msg'
diff --git a/test/integration/targets/ec2_group/tasks/data_validation.yml b/test/integration/targets/ec2_group/tasks/data_validation.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/ec2_group/tasks/data_validation.yml
@@ -0,0 +1,44 @@
+---
+- block:
+    - name: set up aws connection info
+      set_fact:
+        aws_connection_info: &aws_connection_info
+          aws_access_key: "{{ aws_access_key }}"
+          aws_secret_key: "{{ aws_secret_key }}"
+          security_token: "{{ security_token }}"
+          region: "{{ aws_region }}"
+      no_log: yes
+    - name: Create a group with only the default rule
+      ec2_group:
+        name: '{{ec2_group_name}}-input-tests'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+
+    - name: Run through some common weird port specs
+      ec2_group:
+        name: '{{ec2_group_name}}-input-tests'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        rules:
+          - "{{ item }}"
+      with_items:
+        - proto: tcp
+          from_port: "8182"
+          to_port: 8182
+          cidr_ipv6: "64:ff9b::/96"
+          rule_desc: Mixed string and non-string ports
+        - proto: tcp
+          ports:
+          - "9000"
+          - 9001
+          - 9002-9005
+          cidr_ip: "1.2.3.0/24"
+  always:
+    - name: tidy up input testing group
+      ec2_group:
+        name: '{{ec2_group_name}}-input-tests'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        state: absent
+        <<: *aws_connection_info
+      ignore_errors: yes
diff --git a/test/integration/targets/ec2_group/tasks/egress_tests.yml b/test/integration/targets/ec2_group/tasks/egress_tests.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/ec2_group/tasks/egress_tests.yml
@@ -0,0 +1,175 @@
+---
+- block:
+    - name: set up aws connection info
+      set_fact:
+        aws_connection_info: &aws_connection_info
+          aws_access_key: "{{ aws_access_key }}"
+          aws_secret_key: "{{ aws_secret_key }}"
+          security_token: "{{ security_token }}"
+          region: "{{ aws_region }}"
+      no_log: yes
+
+
+    - name: Create a group with only the default rule
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: assert default rule is in place (expected changed=true)
+      assert:
+        that:
+          - result is changed
+          - result.ip_permissions|length == 0
+          - result.ip_permissions_egress|length == 1
+          - result.ip_permissions_egress[0].ip_ranges[0].cidr_ip == '0.0.0.0/0'
+
+    - name: Create a group with only the default rule
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        purge_rules_egress: false
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: assert default rule is not purged (expected changed=false)
+      assert:
+        that:
+          - result is not changed
+          - result.ip_permissions|length == 0
+          - result.ip_permissions_egress|length == 1
+          - result.ip_permissions_egress[0].ip_ranges[0].cidr_ip == '0.0.0.0/0'
+
+    - name: Pass empty egress rules without purging, should leave default rule in place
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        description: '{{ec2_group_description}}'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        purge_rules_egress: false
+        rules_egress: []
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: assert default rule is not purged (expected changed=false)
+      assert:
+        that:
+          - result is not changed
+          - result.ip_permissions|length == 0
+          - result.ip_permissions_egress|length == 1
+          - result.ip_permissions_egress[0].ip_ranges[0].cidr_ip == '0.0.0.0/0'
+
+    - name: Purge rules, including the default
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        description: '{{ec2_group_description}}'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        purge_rules_egress: true
+        rules_egress: []
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: assert default rule is not purged (expected changed=false)
+      assert:
+        that:
+          - result is changed
+          - result.ip_permissions|length == 0
+          - result.ip_permissions_egress|length == 0
+
+    - name: Add a custom egress rule
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        description: '{{ec2_group_description}}'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        rules_egress:
+        - proto: tcp
+          ports:
+          - 1212
+          cidr_ip: 1.2.1.2/32
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: assert first rule is here
+      assert:
+        that:
+          - result.ip_permissions_egress|length == 1
+
+    - name: Add a second custom egress rule
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        description: '{{ec2_group_description}}'
+        purge_rules_egress: false
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        rules_egress:
+        - proto: tcp
+          ports:
+          - 2323
+          cidr_ip: 2.3.2.3/32
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: assert the first rule is not purged
+      assert:
+        that:
+          - result.ip_permissions_egress|length == 2
+
+    - name: Purge the second rule
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        description: '{{ec2_group_description}}'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        rules_egress:
+        - proto: tcp
+          ports:
+          - 1212
+          cidr_ip: 1.2.1.2/32
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: assert first rule is here
+      assert:
+        that:
+          - result.ip_permissions_egress|length == 1
+          - result.ip_permissions_egress[0].ip_ranges[0].cidr_ip == '1.2.1.2/32'
+
+    - name: add a rule for all TCP ports
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        description: '{{ec2_group_description}}'
+        rules_egress:
+        - proto: tcp
+          ports: 0-65535
+          cidr_ip: 0.0.0.0/0
+        <<: *aws_connection_info
+        state: present
+        vpc_id: '{{ vpc_result.vpc.id }}'
+      register: result
+
+    - name: Re-add the default rule
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        description: '{{ec2_group_description}}'
+        rules_egress:
+        - proto: -1
+          cidr_ip: 0.0.0.0/0
+        <<: *aws_connection_info
+        state: present
+        vpc_id: '{{ vpc_result.vpc.id }}'
+      register: result
+  always:
+    - name: tidy up egress rule test security group
+      ec2_group:
+        name: '{{ec2_group_name}}-egress-tests'
+        state: absent
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        <<: *aws_connection_info
+      ignore_errors: yes
diff --git a/test/integration/targets/ec2_group/tasks/ipv6_default_tests.yml b/test/integration/targets/ec2_group/tasks/ipv6_default_tests.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/ec2_group/tasks/ipv6_default_tests.yml
@@ -0,0 +1,103 @@
+---
+- name: set up aws connection info
+  set_fact:
+    aws_connection_info: &aws_connection_info
+      aws_access_key: "{{ aws_access_key }}"
+      aws_secret_key: "{{ aws_secret_key }}"
+      security_token: "{{ security_token }}"
+      region: "{{ aws_region }}"
+  no_log: yes
+# ============================================================
+- name: test state=present for ipv6 (expected changed=true) (CHECK MODE)
+  ec2_group:
+    name: '{{ec2_group_name}}'
+    description: '{{ec2_group_description}}'
+    <<: *aws_connection_info
+    state: present
+    rules:
+    - proto: "tcp"
+      from_port: 8182
+      to_port: 8182
+      cidr_ipv6: "64:ff9b::/96"
+  check_mode: true
+  register: result
+
+- name: assert state=present (expected changed=true)
+  assert:
+    that:
+       - 'result.changed'
+
+# ============================================================
+- name: test state=present for ipv6 (expected changed=true)
+  ec2_group:
+    name: '{{ec2_group_name}}'
+    description: '{{ec2_group_description}}'
+    <<: *aws_connection_info
+    state: present
+    rules:
+    - proto: "tcp"
+      from_port: 8182
+      to_port: 8182
+      cidr_ipv6: "64:ff9b::/96"
+  register: result
+
+- name: assert state=present (expected changed=true)
+  assert:
+    that:
+       - 'result.changed'
+       - 'result.group_id.startswith("sg-")'
+
+# ============================================================
+- name: test rules_egress state=present for ipv6 (expected changed=true) (CHECK MODE)
+  ec2_group:
+    name: '{{ec2_group_name}}'
+    description: '{{ec2_group_description}}'
+    <<: *aws_connection_info
+    state: present
+    rules:
+    - proto: "tcp"
+      from_port: 8182
+      to_port: 8182
+      cidr_ipv6: "64:ff9b::/96"
+    rules_egress:
+    - proto: "tcp"
+      from_port: 8181
+      to_port: 8181
+      cidr_ipv6: "64:ff9b::/96"
+  check_mode: true
+  register: result
+
+- name: assert state=present (expected changed=true)
+  assert:
+    that:
+       - 'result.changed'
+
+# ============================================================
+- name: test rules_egress state=present for ipv6 (expected changed=true)
+  ec2_group:
+    name: '{{ec2_group_name}}'
+    description: '{{ec2_group_description}}'
+    <<: *aws_connection_info
+    state: present
+    rules:
+    - proto: "tcp"
+      from_port: 8182
+      to_port: 8182
+      cidr_ipv6: "64:ff9b::/96"
+    rules_egress:
+    - proto: "tcp"
+      from_port: 8181
+      to_port: 8181
+      cidr_ipv6: "64:ff9b::/96"
+  register: result
+
+- name: assert state=present (expected changed=true)
+  assert:
+    that:
+       - 'result.changed'
+       - 'result.group_id.startswith("sg-")'
+- name: delete it
+  ec2_group:
+    name: '{{ec2_group_name}}'
+    <<: *aws_connection_info
+    state: absent
diff --git a/test/integration/targets/ec2_group/tasks/main.yml b/test/integration/targets/ec2_group/tasks/main.yml
--- a/test/integration/targets/ec2_group/tasks/main.yml
+++ b/test/integration/targets/ec2_group/tasks/main.yml
@@ -8,159 +8,8 @@
 
 # - include: ../../setup_ec2/tasks/common.yml module_name: ec2_group
 
+- include: ./credential_tests.yml
 - block:
-
-    # ============================================================
-    - name: test failure with no parameters
-      ec2_group:
-      register: result
-      ignore_errors: true
-
-    - name: assert failure with no parameters
-      assert:
-        that:
-           - 'result.failed'
-           - 'result.msg == "one of the following is required: name, group_id"'
-
-    # ============================================================
-    - name: test failure with only name
-      ec2_group:
-        name: '{{ec2_group_name}}'
-      register: result
-      ignore_errors: true
-
-    - name: assert failure with only name
-      assert:
-        that:
-           - 'result.failed'
-           - 'result.msg == "Must provide description when state is present."'
-
-    # ============================================================
-    - name: test failure with only description
-      ec2_group:
-        description: '{{ec2_group_description}}'
-      register: result
-      ignore_errors: true
-
-    - name: assert failure with only description
-      assert:
-        that:
-           - 'result.failed'
-           - 'result.msg == "one of the following is required: name, group_id"'
-
-    # ============================================================
-    - name: test failure with empty description (AWS API requires non-empty string desc)
-      ec2_group:
-        name: '{{ec2_group_name}}'
-        description: ''
-        region: '{{ec2_region}}'
-      register: result
-      ignore_errors: true
-
-    - name: assert failure with empty description
-      assert:
-        that:
-           - 'result.failed'
-           - 'result.msg == "Must provide description when state is present."'
-
-    # ============================================================
-    - name: test valid region parameter
-      ec2_group:
-        name: '{{ec2_group_name}}'
-        description: '{{ec2_group_description}}'
-        region: '{{ec2_region}}'
-      register: result
-      ignore_errors: true
-
-    - name: assert valid region parameter
-      assert:
-        that:
-           - 'result.failed'
-           - '"Unable to locate credentials" in result.msg'
-
-    # ============================================================
-    - name: test environment variable EC2_REGION
-      ec2_group:
-        name: '{{ec2_group_name}}'
-        description: '{{ec2_group_description}}'
-      environment:
-        EC2_REGION: '{{ec2_region}}'
-      register: result
-      ignore_errors: true
-
-    - name: assert environment variable EC2_REGION
-      assert:
-        that:
-           - 'result.failed'
-           - '"Unable to locate credentials" in result.msg'
-
-    # ============================================================
-    - name: test invalid ec2_url parameter
-      ec2_group:
-        name: '{{ec2_group_name}}'
-        description: '{{ec2_group_description}}'
-      environment:
-        EC2_URL: bogus.example.com
-      register: result
-      ignore_errors: true
-
-    - name: assert invalid ec2_url parameter
-      assert:
-        that:
-           - 'result.failed'
-           - 'result.msg.startswith("The ec2_group module requires a region")'
-
-    # ============================================================
-    - name: test valid ec2_url parameter
-      ec2_group:
-        name: '{{ec2_group_name}}'
-        description: '{{ec2_group_description}}'
-      environment:
-        EC2_URL: '{{ec2_url}}'
-      register: result
-      ignore_errors: true
-
-    - name: assert valid ec2_url parameter
-      assert:
-        that:
-           - 'result.failed'
-           - 'result.msg.startswith("The ec2_group module requires a region")'
-
-    # ============================================================
-    - name: test credentials from environment
-      ec2_group:
-        name: '{{ec2_group_name}}'
-        description: '{{ec2_group_description}}'
-      environment:
-        EC2_REGION: '{{ec2_region}}'
-        EC2_ACCESS_KEY: bogus_access_key
-        EC2_SECRET_KEY: bogus_secret_key
-      register: result
-      ignore_errors: true
-
-    - name: assert ec2_group with valid ec2_url
-      assert:
-        that:
-           - 'result.failed'
-           - '"validate the provided access credentials" in result.msg'
-
-    # ============================================================
-    - name: test credential parameters
-      ec2_group:
-        name: '{{ec2_group_name}}'
-        description: '{{ec2_group_description}}'
-        ec2_region: '{{ec2_region}}'
-        ec2_access_key: 'bogus_access_key'
-        ec2_secret_key: 'bogus_secret_key'
-      register: result
-      ignore_errors: true
-
-    - name: assert credential parameters
-      assert:
-        that:
-           - 'result.failed'
-           - '"validate the provided access credentials" in result.msg'
-
     # ============================================================
     - name: set up aws connection info
       set_fact:
@@ -193,8 +42,26 @@
           Name: "{{ resource_prefix }}-vpc"
           Description: "Created by ansible-test"
       register: vpc_result
+    - include: ./rule_group_create.yml
+    - include: ./egress_tests.yml
+    - include: ./data_validation.yml
 
     # ============================================================
+    - name: test state=absent (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: absent
+      check_mode: true
+      register: result
+
+    - name: assert no changes would be made
+      assert:
+        that:
+          - not result.changed
+
+    # ===========================================================
     - name: test state=absent
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -203,6 +70,21 @@
         state: absent
       register: result
 
+    # ============================================================
+    - name: test state=present (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+      check_mode: true
+      register: result
+
+    - name: assert state=present (expected changed=true)
+      assert:
+        that:
+           - 'result.changed'
+
     # ============================================================
     - name: test state=present (expected changed=true)
       ec2_group:
@@ -218,6 +100,21 @@
            - 'result.changed'
            - 'result.group_id.startswith("sg-")'
 
+    # ============================================================
+    - name: test state=present different description (expected changed=false) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}CHANGED'
+        <<: *aws_connection_info
+        state: present
+      check_mode: true
+      register: result
+
+    - name: assert state=present (expected changed=false)
+      assert:
+        that:
+           - 'not result.changed'
+
     # ============================================================
     - name: test state=present different description (expected changed=false)
       ec2_group:
@@ -251,20 +148,36 @@
 
     # ============================================================
     - name: tests IPv6 with the default VPC
+      include: ./ipv6_default_tests.yml
+      when: default_vpc
+
+    - name: test IPv6 with a specified VPC
       block:
 
         # ============================================================
-        - name: test state=present for ipv6 (expected changed=true)
+        - name: test state=present (expected changed=true) (CHECK MODE)
           ec2_group:
-            name: '{{ec2_group_name}}'
-            description: '{{ec2_group_description}}'
+            name: '{{ ec2_group_name }}-2'
+            description: '{{ ec2_group_description }}-2'
+            state: present
+            vpc_id: '{{ vpc_result.vpc.id }}'
             <<: *aws_connection_info
+          check_mode: true
+          register: result
+
+        - name: assert state=present (expected changed=true)
+          assert:
+            that:
+               - 'result.changed'
+
+        # ============================================================
+        - name: test state=present (expected changed=true)
+          ec2_group:
+            name: '{{ ec2_group_name }}-2'
+            description: '{{ ec2_group_description }}-2'
             state: present
-            rules:
-            - proto: "tcp"
-              from_port: 8182
-              to_port: 8182
-              cidr_ipv6: "64:ff9b::/96"
+            vpc_id: '{{ vpc_result.vpc.id }}'
+            <<: *aws_connection_info
           register: result
 
         - name: assert state=present (expected changed=true)
@@ -274,42 +187,38 @@
                - 'result.group_id.startswith("sg-")'
 
         # ============================================================
-        - name: test rules_egress state=present for ipv6 (expected changed=true)
+        - name: test state=present for ipv6 (expected changed=true) (CHECK MODE)
           ec2_group:
-            name: '{{ec2_group_name}}'
-            description: '{{ec2_group_description}}'
-            <<: *aws_connection_info
+            name: '{{ ec2_group_name }}-2'
+            description: '{{ ec2_group_description }}-2'
             state: present
+            vpc_id: '{{ vpc_result.vpc.id }}'
             rules:
             - proto: "tcp"
               from_port: 8182
               to_port: 8182
               cidr_ipv6: "64:ff9b::/96"
-            rules_egress:
-            - proto: "tcp"
-              from_port: 8181
-              to_port: 8181
-              cidr_ipv6: "64:ff9b::/96"
+            <<: *aws_connection_info
+          check_mode: true
           register: result
 
         - name: assert state=present (expected changed=true)
           assert:
             that:
                - 'result.changed'
-               - 'result.group_id.startswith("sg-")'
-
-      when: default_vpc
-
-    - name: test IPv6 with a specified VPC
-      block:
 
         # ============================================================
-        - name: test state=present (expected changed=true)
+        - name: test state=present for ipv6 (expected changed=true)
           ec2_group:
             name: '{{ ec2_group_name }}-2'
             description: '{{ ec2_group_description }}-2'
             state: present
             vpc_id: '{{ vpc_result.vpc.id }}'
+            rules:
+            - proto: "tcp"
+              from_port: 8182
+              to_port: 8182
+              cidr_ipv6: "64:ff9b::/96"
             <<: *aws_connection_info
           register: result
 
@@ -320,7 +229,7 @@
                - 'result.group_id.startswith("sg-")'
 
         # ============================================================
-        - name: test state=present for ipv6 (expected changed=true)
+        - name: test state=present for ipv6 (expected changed=false) (CHECK MODE)
           ec2_group:
             name: '{{ ec2_group_name }}-2'
             description: '{{ ec2_group_description }}-2'
@@ -332,17 +241,16 @@
               to_port: 8182
               cidr_ipv6: "64:ff9b::/96"
             <<: *aws_connection_info
+          check_mode: true
           register: result
 
-        - name: assert state=present (expected changed=true)
+        - name: assert nothing changed
           assert:
             that:
-               - 'result.changed'
-               - 'result.group_id.startswith("sg-")'
+              - 'not result.changed'
 
         # ============================================================
-
-        - name: test state=present for ipv6 (expected changed=true)
+        - name: test state=present for ipv6 (expected changed=false)
           ec2_group:
             name: '{{ ec2_group_name }}-2'
             description: '{{ ec2_group_description }}-2'
@@ -361,6 +269,32 @@
             that:
               - 'not result.changed'
 
+        # ============================================================
+        - name: test rules_egress state=present for ipv6 (expected changed=true) (CHECK MODE)
+          ec2_group:
+            name: '{{ ec2_group_name }}-2'
+            description: '{{ ec2_group_description }}-2'
+            state: present
+            vpc_id: '{{ vpc_result.vpc.id }}'
+            rules:
+            - proto: "tcp"
+              from_port: 8182
+              to_port: 8182
+              cidr_ipv6: "64:ff9b::/96"
+            rules_egress:
+            - proto: "tcp"
+              from_port: 8181
+              to_port: 8181
+              cidr_ipv6: "64:ff9b::/96"
+            <<: *aws_connection_info
+          check_mode: true
+          register: result
+
+        - name: assert state=present (expected changed=true)
+          assert:
+            that:
+               - 'result.changed'
+
         # ============================================================
         - name: test rules_egress state=present for ipv6 (expected changed=true)
           ec2_group:
@@ -388,7 +322,22 @@
                - 'result.group_id.startswith("sg-")'
 
         # ============================================================
+        - name: test state=absent (expected changed=true) (CHECK MODE)
+          ec2_group:
+            name: '{{ ec2_group_name }}-2'
+            description: '{{ ec2_group_description }}-2'
+            state: absent
+            vpc_id: '{{ vpc_result.vpc.id }}'
+            <<: *aws_connection_info
+          check_mode: true
+          register: result
 
+        - name: assert group was removed
+          assert:
+            that:
+              - 'result.changed'
+
+        # ============================================================
         - name: test state=absent (expected changed=true)
           ec2_group:
             name: '{{ ec2_group_name }}-2'
@@ -403,12 +352,30 @@
             that:
               - 'result.changed'
 
+    # ============================================================
+    - name: test state=present for ipv4 (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        rules:
+        - proto: "tcp"
+          from_port: 8182
+          to_port: 8182
+          cidr_ip: "1.1.1.1/32"
+      check_mode: true
+      register: result
+
+    - name: assert state=present (expected changed=true)
+      assert:
+        that:
+        - 'result.changed'
+
     # ============================================================
     - name: test state=present for ipv4 (expected changed=true)
       ec2_group:
         name: '{{ec2_group_name}}'
         description: '{{ec2_group_description}}'
-        ec2_region: '{{ec2_region}}'
         <<: *aws_connection_info
         rules:
         - proto: "tcp"
@@ -425,6 +392,22 @@
         - 'result.ip_permissions|length == 1'
         - 'result.ip_permissions_egress|length == 1'
 
+    # ============================================================
+    - name: add same rule to the existing group  (expected changed=false) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+        rules:
+        - proto: "tcp"
+          from_port: 8182
+          to_port: 8182
+          cidr_ip: "1.1.1.1/32"
+      check_mode: true
+      register: check_result
+
+
     # ============================================================
     - name: add same rule to the existing group  (expected changed=false)
       ec2_group:
@@ -445,6 +428,33 @@
           - 'not result.changed'
           - 'result.group_id.startswith("sg-")'
 
+    - name: assert state=present (expected changed=false)
+      assert:
+        that:
+          - 'not check_result.changed'
+
+    # ============================================================
+    - name: add a rule that auto creates another security group (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+        purge_rules: no
+        rules:
+        - proto: "tcp"
+          group_name: "{{ resource_prefix }} - Another security group"
+          group_desc: Another security group
+          ports: 7171
+      check_mode: true
+      register: result
+
+    - name: check that there are now two rules
+      assert:
+        that:
+          - result.changed
+
+    # ============================================================
     - name: add a rule that auto creates another security group
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -468,6 +478,31 @@
             result.ip_permissions[1].user_id_group_pairs
           - 'result.ip_permissions_egress[0].ip_protocol == "-1"'
 
+    # ============================================================
+    - name: test ip rules convert port numbers from string to int (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+        rules:
+        - proto: "tcp"
+          from_port: "8183"
+          to_port: "8183"
+          cidr_ip: "1.1.1.1/32"
+        rules_egress:
+        - proto: "tcp"
+          from_port: "8184"
+          to_port: "8184"
+          cidr_ip: "1.1.1.1/32"
+      check_mode: true
+      register: result
+
+    - name: assert state=present (expected changed=true)
+      assert:
+        that:
+           - 'result.changed'
+
     # ============================================================
     - name: test ip rules convert port numbers from string to int (expected changed=true)
       ec2_group:
@@ -496,6 +531,31 @@
            - 'result.ip_permissions_egress[0].ip_protocol == "tcp"'
 
 
+    # ============================================================
+    - name: test group rules convert port numbers from string to int (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+        rules:
+        - proto: "tcp"
+          from_port: "8185"
+          to_port: "8185"
+          group_id: "{{result.group_id}}"
+        rules_egress:
+        - proto: "tcp"
+          from_port: "8186"
+          to_port: "8186"
+          group_id: "{{result.group_id}}"
+      check_mode: true
+      register: result
+
+    - name: assert state=present (expected changed=true)
+      assert:
+        that:
+           - 'result.changed'
+
     # ============================================================
     - name: test group rules convert port numbers from string to int (expected changed=true)
       ec2_group:
@@ -521,9 +581,30 @@
            - 'result.changed'
            - 'result.group_id.startswith("sg-")'
 
-
     # ============================================================
+    - name: test adding a range of ports and ports given as strings (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+        # set purge_rules to false so we don't get a false positive from previously added rules
+        purge_rules: false
+        rules:
+        - proto: "tcp"
+          ports:
+            - 8183-8190
+            - '8192'
+          cidr_ip: 1.1.1.1/32
+      check_mode: true
+      register: result
+
+    - name: assert state=present (expected changed=true)
+      assert:
+        that:
+          - 'result.changed'
 
+    # ============================================================
     - name: test adding a range of ports and ports given as strings (expected changed=true)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -547,7 +628,28 @@
           - 'result.group_id.startswith("sg-")'
 
     # ============================================================
+    - name: test adding a rule with a IPv4 CIDR with host bits set (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+        # set purge_rules to false so we don't get a false positive from previously added rules
+        purge_rules: false
+        rules:
+        - proto: "tcp"
+          ports:
+            - 8195
+          cidr_ip: 10.0.0.1/8
+      check_mode: true
+      register: result
 
+    - name: assert state=present (expected changed=true)
+      assert:
+        that:
+          - 'result.changed'
+
+    # ============================================================
     - name: test adding a rule with a IPv4 CIDR with host bits set (expected changed=true)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -570,7 +672,23 @@
           - 'result.group_id.startswith("sg-")'
 
     # ============================================================
+    - name: test adding the same rule with a IPv4 CIDR with host bits set (expected changed=false) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+        # set purge_rules to false so we don't get a false positive from previously added rules
+        purge_rules: false
+        rules:
+        - proto: "tcp"
+          ports:
+            - 8195
+          cidr_ip: 10.0.0.1/8
+      check_mode: true
+      register: check_result
 
+    # ============================================================
     - name: test adding the same rule with a IPv4 CIDR with host bits set (expected changed=false and a warning)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -586,6 +704,11 @@
           cidr_ip: 10.0.0.1/8
       register: result
 
+    - name: assert state=present (expected changed=false and a warning)
+      assert:
+        that:
+          - 'not check_result.changed'
+
     - name: assert state=present (expected changed=false and a warning)
       assert:
         that:
@@ -593,18 +716,37 @@
           - 'not result.changed'
           - 'result.group_id.startswith("sg-")'
 
-    # ============================================================
-    - name: test using the default VPC
-      block:
+    # ============================================================
+    - name: test using the default VPC
+      block:
+
+        - name: test adding a rule with a IPv6 CIDR with host bits set (expected changed=true) (CHECK MODE)
+          ec2_group:
+            name: '{{ec2_group_name}}'
+            description: '{{ec2_group_description}}'
+            <<: *aws_connection_info
+            state: present
+            # set purge_rules to false so we don't get a false positive from previously added rules
+            purge_rules: false
+            rules:
+            - proto: "tcp"
+              ports:
+                - 8196
+              cidr_ipv6: '2001:db00::1/24'
+          check_mode: true
+          register: result
+
+        - name: assert state=present (expected changed=true)
+          assert:
+            that:
+              - 'result.changed'
 
+        # ============================================================
         - name: test adding a rule with a IPv6 CIDR with host bits set (expected changed=true)
           ec2_group:
             name: '{{ec2_group_name}}'
             description: '{{ec2_group_description}}'
-            ec2_region: '{{ec2_region}}'
-            ec2_access_key: '{{ec2_access_key}}'
-            ec2_secret_key: '{{ec2_secret_key}}'
-            security_token: '{{security_token}}'
+            <<: *aws_connection_info
             state: present
             # set purge_rules to false so we don't get a false positive from previously added rules
             purge_rules: false
@@ -647,6 +789,20 @@
 
       when: default_vpc
 
+    # ============================================================
+    - name: test state=absent (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        state: absent
+        <<: *aws_connection_info
+      check_mode: true
+      register: result
+
+    - name: assert state=absent (expected changed=true)
+      assert:
+        that:
+           - 'result.changed'
+
     # ============================================================
     - name: test state=absent (expected changed=true)
       ec2_group:
@@ -661,6 +817,28 @@
            - 'result.changed'
            - 'not result.group_id'
 
+    # ============================================================
+    - name: create security group in the VPC (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        state: present
+        rules:
+        - proto: "tcp"
+          from_port: 8182
+          to_port: 8182
+          cidr_ip: "1.1.1.1/32"
+      check_mode: true
+      register: result
+
+    - name: assert state=present (expected changed=true)
+      assert:
+        that:
+          - 'result.changed'
+
+    # ============================================================
     - name: create security group in the VPC
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -683,7 +861,30 @@
           - 'result.group_id.startswith("sg-")'
 
     # ============================================================
+    - name: test adding tags (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        state: present
+        rules:
+        - proto: "tcp"
+          from_port: 8182
+          to_port: 8182
+          cidr_ip: "1.1.1.1/32"
+        tags:
+          tag1: test1
+          tag2: test2
+      check_mode: true
+      register: result
+
+    - name: assert that tags were added (expected changed=true)
+      assert:
+        that:
+          - 'result.changed'
 
+    # ============================================================
     - name: test adding tags (expected changed=true)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -699,7 +900,6 @@
         tags:
           tag1: test1
           tag2: test2
-
       register: result
 
     - name: assert that tags were added (expected changed=true)
@@ -709,7 +909,31 @@
           - 'result.tags == {"tag1": "test1", "tag2": "test2"}'
 
     # ============================================================
+    - name: test that tags are present (expected changed=False) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        state: present
+        purge_rules_egress: false
+        rules:
+        - proto: "tcp"
+          from_port: 8182
+          to_port: 8182
+          cidr_ip: "1.1.1.1/32"
+        tags:
+          tag1: test1
+          tag2: test2
+      check_mode: true
+      register: result
 
+    - name: assert that tags were not changed (expected changed=False)
+      assert:
+        that:
+          - 'not result.changed'
+
+    # ============================================================
     - name: test that tags are present (expected changed=False)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -735,7 +959,29 @@
           - 'result.tags == {"tag1": "test1", "tag2": "test2"}'
 
     # ============================================================
+    - name: test purging tags (expected changed=True) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        state: present
+        rules:
+        - proto: "tcp"
+          from_port: 8182
+          to_port: 8182
+          cidr_ip: "1.1.1.1/32"
+        tags:
+          tag1: test1
+      check_mode: true
+      register: result
+
+    - name: assert that tag2 was removed (expected changed=true)
+      assert:
+        that:
+          - 'result.changed'
 
+    # ============================================================
     - name: test purging tags (expected changed=True)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -804,7 +1050,47 @@
           - 'not result.tags'
 
     # ============================================================
+    - name: test adding a rule and egress rule descriptions (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        # purge the other rules so assertions work for the subsequent tests for rule descriptions
+        purge_rules_egress: true
+        purge_rules: true
+        state: present
+        rules:
+        - proto: "tcp"
+          ports:
+          - 8281
+          cidr_ipv6: 1001:d00::/24
+          rule_desc: ipv6 rule desc 1
+        rules_egress:
+        - proto: "tcp"
+          ports:
+          - 8282
+          cidr_ip: 2.2.2.2/32
+          rule_desc: egress rule desc 1
+      check_mode: true
+      register: result
 
+    - name: assert that rule descriptions are created (expected changed=true)
+      # Only assert this if rule description is defined as the botocore version may < 1.7.2.
+      # It's still helpful to have these tests run on older versions since it verifies backwards
+      # compatibility with this feature.
+      assert:
+        that:
+          - 'result.changed'
+      when: result.ip_permissions_egress[0].ip_ranges[0].description is defined
+
+    - name: if an older version of botocore is installed changes should still have changed due to purged rules (expected changed=true)
+      assert:
+        that:
+          - 'result.changed'
+      when: result.ip_permissions_egress[0].ip_ranges[0].description is undefined
+
+    # ============================================================
     - name: test adding a rule and egress rule descriptions (expected changed=true)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -847,7 +1133,46 @@
       when: result.ip_permissions_egress[0].ip_ranges[0].description is undefined
 
     # ============================================================
+    - name: test modifying rule and egress rule descriptions (expected changed=true) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        purge_rules_egress: false
+        purge_rules: false
+        state: present
+        rules:
+        - proto: "tcp"
+          ports:
+          - 8281
+          cidr_ipv6: 1001:d00::/24
+          rule_desc: ipv6 rule desc 2
+        rules_egress:
+        - proto: "tcp"
+          ports:
+          - 8282
+          cidr_ip: 2.2.2.2/32
+          rule_desc: egress rule desc 2
+      check_mode: true
+      register: result
+
+    - name: assert that rule descriptions were modified (expected changed=true)
+      # Only assert this if rule description is defined as the botocore version may < 1.7.2.
+      # It's still helpful to have these tests run on older versions since it verifies backwards
+      # compatibility with this feature.
+      assert:
+        that:
+          - 'result.changed'
+      when: result.ip_permissions_egress[0].ip_ranges[0].description is defined
+
+    - name: if an older version of botocore is installed everything should stay the same (expected changed=false)
+      assert:
+        that:
+          - 'not result.changed'
+      when: result.ip_permissions_egress[0].ip_ranges[0].description is undefined and result.ip_permissions_egress[1].ip_ranges[0].description is undefined
 
+    # ============================================================
     - name: test modifying rule and egress rule descriptions (expected changed=true)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -921,7 +1246,46 @@
           - 'result.ip_permissions_egress|length == 1'
 
     # ============================================================
+    - name: test that keeping the same rule descriptions (expected changed=false) (CHECK MODE)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        purge_rules_egress: false
+        purge_rules: false
+        state: present
+        rules:
+        - proto: "tcp"
+          ports:
+          - 8281
+          cidr_ipv6: 1001:d00::/24
+          rule_desc: ipv6 rule desc 2
+        rules_egress:
+        - proto: "tcp"
+          ports:
+          - 8282
+          cidr_ip: 2.2.2.2/32
+          rule_desc: egress rule desc 2
+      check_mode: true
+      register: result
+
+    - name: assert that rule descriptions stayed the same (expected changed=false)
+      # Only assert this if rule description is defined as the botocore version may < 1.7.2.
+      # It's still helpful to have these tests run on older versions since it verifies backwards
+      # compatibility with this feature.
+      assert:
+        that:
+          - 'not result.changed'
+      when: result.ip_permissions_egress[0].ip_ranges[0].description is defined
+
+    - name: if an older version of botocore is installed everything should stay the same (expected changed=false)
+      assert:
+        that:
+          - 'not result.changed'
+      when: result.ip_permissions_egress[0].ip_ranges[0].description is undefined
 
+    # ============================================================
     - name: test that keeping the same rule descriptions (expected changed=false)
       ec2_group:
         name: '{{ec2_group_name}}'
@@ -963,8 +1327,7 @@
       when: result.ip_permissions_egress[0].ip_ranges[0].description is undefined
 
     # ============================================================
-
-    - name: test removing rule descriptions (expected changed=true)
+    - name: test removing rule descriptions (expected changed=true) (CHECK MODE)
       ec2_group:
         name: '{{ec2_group_name}}'
         description: '{{ec2_group_description}}'
@@ -985,6 +1348,7 @@
           - 8282
           cidr_ip: 2.2.2.2/32
           rule_desc:
+      check_mode: true
       register: result
 
     - name: assert that rule descriptions were removed (expected changed=true)
@@ -994,8 +1358,6 @@
       assert:
         that:
           - 'result.changed'
-          - 'not result.ip_permissions[0].ipv6_ranges[0].description'
-          - 'not result.ip_permissions_egress[0].ip_ranges[0].description'
       when: result.ip_permissions_egress[0].ip_ranges[0].description is defined
 
     - name: if an older version of botocore is installed everything should stay the same (expected changed=false)
@@ -1004,6 +1366,47 @@
           - 'not result.changed'
       when: result.ip_permissions_egress[0].ip_ranges[0].description is undefined
 
+    # ============================================================
+    - name: test removing rule descriptions (expected changed=true)
+      ec2_group:
+        name: '{{ec2_group_name}}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        purge_rules_egress: false
+        purge_rules: false
+        state: present
+        rules:
+        - proto: "tcp"
+          ports:
+          - 8281
+          cidr_ipv6: 1001:d00::/24
+          rule_desc:
+        rules_egress:
+        - proto: "tcp"
+          ports:
+          - 8282
+          cidr_ip: 2.2.2.2/32
+          rule_desc:
+      register: result
+      ignore_errors: true
+
+    - name: assert that rule descriptions were removed (expected changed=true with newer botocore)
+      # Only assert this if rule description is defined as the botocore version may < 1.7.2.
+      # It's still helpful to have these tests run on older versions since it verifies backwards
+      # compatibility with this feature.
+      assert:
+        that:
+          - 'result.ip_permissions[0].ipv6_ranges[0].description is undefined'
+          - 'result.ip_permissions_egress[0].ip_ranges[0].description is undefined'
+      when: result is changed
+
+    - name: if an older version of botocore is installed everything should stay the same (expected changed=false)
+      assert:
+        that:
+          - 'not result.changed'
+      when: result.failed
+
     # ============================================================
 
     - name: test state=absent (expected changed=true)
@@ -1020,7 +1423,6 @@
            - 'not result.group_id'
 
   always:
-
     # ============================================================
     - name: tidy up security group
       ec2_group:
diff --git a/test/integration/targets/ec2_group/tasks/rule_group_create.yml b/test/integration/targets/ec2_group/tasks/rule_group_create.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/ec2_group/tasks/rule_group_create.yml
@@ -0,0 +1,132 @@
+---
+- block:
+    - name: set up aws connection info
+      set_fact:
+        aws_connection_info: &aws_connection_info
+          aws_access_key: "{{ aws_access_key }}"
+          aws_secret_key: "{{ aws_secret_key }}"
+          security_token: "{{ security_token }}"
+          region: "{{ aws_region }}"
+      no_log: yes
+
+    - name: Create a group with self-referring rule
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-1'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        rules:
+        - proto: "tcp"
+          from_port: 8000
+          to_port: 8100
+          group_name: '{{ec2_group_name}}-auto-create-1'
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: Create a second group rule
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-2'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+
+    - name: Create a series of rules with a recently created group as target
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-1'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        purge_rules: false
+        rules:
+        - proto: "tcp"
+          from_port: "{{ item }}"
+          to_port: "{{ item }}"
+          group_name: '{{ec2_group_name}}-auto-create-2'
+        <<: *aws_connection_info
+        state: present
+      register: result
+      with_items:
+        - 20
+        - 40
+        - 60
+        - 80
+
+    - name: Create a group with only the default rule
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-1'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        rules:
+        - proto: "tcp"
+          from_port: 8182
+          to_port: 8182
+          group_name: '{{ec2_group_name}}-auto-create-3'
+        <<: *aws_connection_info
+        state: present
+      register: result
+      ignore_errors: true
+
+    - name: assert you can't create a new group from a rule target with no description
+      assert:
+        that:
+          - result is failed
+
+    - name: Create a group with a target of a separate group
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-1'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        rules:
+        - proto: tcp
+          ports:
+            - 22
+            - 80
+          group_name: '{{ec2_group_name}}-auto-create-3'
+          group_desc: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+      register: result
+
+    - name: Create a 4th group
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-4'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        <<: *aws_connection_info
+        state: present
+        rules:
+        - proto: tcp
+          ports:
+            - 22
+          cidr_ip: 0.0.0.0/0
+
+    - name: use recently created group in a rule
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-5'
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        description: '{{ec2_group_description}}'
+        rules:
+        - proto: tcp
+          ports:
+            - 443
+          group_name: '{{ec2_group_name}}-auto-create-4'
+        <<: *aws_connection_info
+        state: present
+
+  always:
+    - name: tidy up egress rule test security group
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-{{ item }}'
+        state: absent
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        <<: *aws_connection_info
+      ignore_errors: yes
+      with_items: [5, 4, 3, 2, 1]
+    - name: tidy up egress rule test security group
+      ec2_group:
+        name: '{{ec2_group_name}}-auto-create-{{ item }}'
+        state: absent
+        vpc_id: '{{ vpc_result.vpc.id }}'
+        <<: *aws_connection_info
+      ignore_errors: yes
+      with_items: [1, 2, 3, 4, 5]
diff --git a/test/sanity/pylint/ignore.txt b/test/sanity/pylint/ignore.txt
--- a/test/sanity/pylint/ignore.txt
+++ b/test/sanity/pylint/ignore.txt
@@ -2,7 +2,6 @@ lib/ansible/module_utils/network/iosxr/iosxr.py ansible-format-automatic-specifi
 lib/ansible/modules/cloud/amazon/aws_api_gateway.py ansible-format-automatic-specification
 lib/ansible/modules/cloud/amazon/aws_kms.py ansible-format-automatic-specification
 lib/ansible/modules/cloud/amazon/ec2_eip.py ansible-format-automatic-specification
-lib/ansible/modules/cloud/amazon/ec2_group.py ansible-format-automatic-specification
 lib/ansible/modules/cloud/amazon/ecs_ecr.py ansible-format-automatic-specification
 lib/ansible/modules/cloud/amazon/sns.py ansible-format-automatic-specification
 lib/ansible/modules/cloud/azure/azure_rm_acs.py ansible-format-automatic-specification
diff --git a/test/units/modules/cloud/amazon/test_ec2_group.py b/test/units/modules/cloud/amazon/test_ec2_group.py
new file mode 100644
--- /dev/null
+++ b/test/units/modules/cloud/amazon/test_ec2_group.py
@@ -0,0 +1,80 @@
+import pytest
+
+from ansible.modules.cloud.amazon import ec2_group as group_module
+
+
+def test_from_permission():
+    internal_http = {
+        u'FromPort': 80,
+        u'IpProtocol': 'tcp',
+        u'IpRanges': [
+            {
+                u'CidrIp': '10.0.0.0/8',
+                u'Description': 'Foo Bar Baz'
+            },
+        ],
+        u'Ipv6Ranges': [
+            {u'CidrIpv6': 'fe80::94cc:8aff:fef6:9cc/64'},
+        ],
+        u'PrefixListIds': [],
+        u'ToPort': 80,
+        u'UserIdGroupPairs': [],
+    }
+    perms = list(group_module.rule_from_group_permission(internal_http))
+    assert len(perms) == 2
+    assert perms[0].target == '10.0.0.0/8'
+    assert perms[0].target_type == 'ipv4'
+    assert perms[0].description == 'Foo Bar Baz'
+    assert perms[1].target == 'fe80::94cc:8aff:fef6:9cc/64'
+
+    global_egress = {
+        'IpProtocol': '-1',
+        'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
+        'Ipv6Ranges': [],
+        'PrefixListIds': [],
+        'UserIdGroupPairs': []
+    }
+    perms = list(group_module.rule_from_group_permission(global_egress))
+    assert len(perms) == 1
+    assert perms[0].target == '0.0.0.0/0'
+    assert perms[0].port_range == (None, None)
+
+    internal_prefix_http = {
+        u'FromPort': 80,
+        u'IpProtocol': 'tcp',
+        u'PrefixListIds': [
+            {'PrefixListId': 'p-1234'}
+        ],
+        u'ToPort': 80,
+        u'UserIdGroupPairs': [],
+    }
+    perms = list(group_module.rule_from_group_permission(internal_prefix_http))
+    assert len(perms) == 1
+    assert perms[0].target == 'p-1234'
+
+
+def test_rule_to_permission():
+    tests = [
+        group_module.Rule((22, 22), 'udp', 'sg-1234567890', 'group', None),
+        group_module.Rule((1, 65535), 'tcp', '0.0.0.0/0', 'ipv4', "All TCP from everywhere"),
+        group_module.Rule((443, 443), 'tcp', 'ip-123456', 'ip_prefix', "Traffic to privatelink IPs"),
+        group_module.Rule((443, 443), 'tcp', 'feed:dead:::beef/64', 'ipv6', None),
+    ]
+    for test in tests:
+        perm = group_module.to_permission(test)
+        assert perm['FromPort'], perm['ToPort'] == test.port_range
+        assert perm['IpProtocol'] == test.protocol
+
+
+def test_validate_ip():
+    class Warner(object):
+        def warn(self, msg):
+            return
+    ips = [
+        ('1.1.1.1/24', '1.1.1.0/24'),
+        ('192.168.56.101/16', '192.168.0.0/16'),
+        ('1203:8fe0:fe80:b897:8990:8a7c:99bf:323d/64', '1203:8fe0:fe80::/64'),
+    ]
+
+    for ip, net in ips:
+        assert group_module.validate_ip(Warner(), ip) == net