Repository: ansible/ansible

Issue:
ec2 module attempting to terminate unrelated instance outside of inventory when using exact_count, instance_tags, and count_tag
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
ec2

##### ANSIBLE VERSION
```
ansible 2.3.0 (devel f1b5dde4cb) last updated 2016/12/16 05:20:05 (GMT -400)
  config file = /home/ecms/horizons_v2/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```

##### CONFIGURATION
N/A

##### OS / ENVIRONMENT
N/A

##### SUMMARY
When iterating over inventory to launch a set of role instances with `exact_count`, `instance_tags`, `count_tag` all set, and no matching instances running, Ansible is attempting to terminate a completely unrelated instance in another VPC and subnet (the instance running Ansible). I was lucky that I had termination protection enabled on that instance.

##### STEPS TO REPRODUCE

```yaml
  - name: Launch instance
    ec2:
      aws_access_key: "{{ aws.access_key }}"
      aws_secret_key: "{{ aws.secret_key }}"
      instance_tags: {"Name":"{{ inventory_hostname }}"}
      exact_count: 1
      count_tag: Name
      key_name: "{{ instance.key_name }}"
      group: "{{ instance.security_group }}"
      instance_type: "{{ instance.type }}"
      image: "{{ instance.image }}"
      wait: true
      region: "{{ aws.region }}"
      vpc_subnet_id: "{{ instance.vpc_subnet_id }}"
      assign_public_ip: "{{ instance.public_ip }}"
      instance_profile_name: "{{ aws.iam_role }}"
      source_dest_check: "{{ instance.source_dest_check }}"
      volumes: "{{ instance.volumes }}"
    register: ec2
```

##### EXPECTED RESULTS
I expect Ansible to only work with the instances that are defined in my inventory, it SHOULD NOT be touching anything outside of that inventory.

##### ACTUAL RESULTS
Instead, Ansible is spitting out a fatal error for each instance in my inventory, but referencing the ID of the unrelated instance in the other VPC/subnet, illustrated below with `i-aaaaaaaa`.

```
fatal: [host.domain.internal]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "assign_public_ip": true,
            "aws_access_key": "*************************",
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "count": 1,
            "count_tag": "Name",
            "ebs_optimized": false,
            "ec2_url": null,
            "exact_count": 1,
            "group": [
                "loadbalancer"
            ],
            "group_id": null,
            "id": null,
            "image": "ami-b73b63a0",
            "instance_ids": null,
            "instance_initiated_shutdown_behavior": null,
            "instance_profile_name": "server",
            "instance_tags": {
                "Name": "host.domain.internal"
            },
            "instance_type": "c3.large",
            "kernel": null,
            "key_name": "host",
            "monitoring": false,
            "network_interfaces": null,
            "placement_group": null,
            "private_ip": null,
            "profile": null,
            "ramdisk": null,
            "region": "us-east-1",
            "security_token": null,
            "source_dest_check": true,
            "spot_launch_group": null,
            "spot_price": null,
            "spot_type": "one-time",
            "spot_wait_timeout": "600",
            "state": "present",
            "tenancy": "default",
            "termination_protection": null,
            "user_data": null,
            "validate_certs": true,
            "volumes": [
                {
                    "delete_on_termination": true,
                    "device_name": "/dev/xvda",
                    "volume_size": 10,
                    "volume_type": "gp2"
                }
            ],
            "vpc_subnet_id": "subnet-********",
            "wait": true,
            "wait_timeout": "300",
            "zone": null
        },
        "module_name": "ec2"
    },
    "msg": "Unable to terminate instance i-aaaaaaaa, error: EC2ResponseError: 400 Bad Request\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Response><Errors><Error><Code>OperationNotPermitted</Code><Message>The instance 'i-aaaaaaaa' may not be terminated. Modify its 'disableApiTermination' instance attribute and try again.</Message></Error></Errors><RequestID>d3a71576-0376-4293-b5ec-9ce9d241fee1</RequestID></Response>"
```

ec2 module attempting to terminate unrelated instance outside of inventory when using exact_count, instance_tags, and count_tag
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
ec2

##### ANSIBLE VERSION
```
ansible 2.3.0 (devel f1b5dde4cb) last updated 2016/12/16 05:20:05 (GMT -400)
  config file = /home/ecms/horizons_v2/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```

##### CONFIGURATION
N/A

##### OS / ENVIRONMENT
N/A

##### SUMMARY
When iterating over inventory to launch a set of role instances with `exact_count`, `instance_tags`, `count_tag` all set, and no matching instances running, Ansible is attempting to terminate a completely unrelated instance in another VPC and subnet (the instance running Ansible). I was lucky that I had termination protection enabled on that instance.

##### STEPS TO REPRODUCE

```yaml
  - name: Launch instance
    ec2:
      aws_access_key: "{{ aws.access_key }}"
      aws_secret_key: "{{ aws.secret_key }}"
      instance_tags: {"Name":"{{ inventory_hostname }}"}
      exact_count: 1
      count_tag: Name
      key_name: "{{ instance.key_name }}"
      group: "{{ instance.security_group }}"
      instance_type: "{{ instance.type }}"
      image: "{{ instance.image }}"
      wait: true
      region: "{{ aws.region }}"
      vpc_subnet_id: "{{ instance.vpc_subnet_id }}"
      assign_public_ip: "{{ instance.public_ip }}"
      instance_profile_name: "{{ aws.iam_role }}"
      source_dest_check: "{{ instance.source_dest_check }}"
      volumes: "{{ instance.volumes }}"
    register: ec2
```

##### EXPECTED RESULTS
I expect Ansible to only work with the instances that are defined in my inventory, it SHOULD NOT be touching anything outside of that inventory.

##### ACTUAL RESULTS
Instead, Ansible is spitting out a fatal error for each instance in my inventory, but referencing the ID of the unrelated instance in the other VPC/subnet, illustrated below with `i-aaaaaaaa`.

```
fatal: [host.domain.internal]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "assign_public_ip": true,
            "aws_access_key": "*************************",
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "count": 1,
            "count_tag": "Name",
            "ebs_optimized": false,
            "ec2_url": null,
            "exact_count": 1,
            "group": [
                "loadbalancer"
            ],
            "group_id": null,
            "id": null,
            "image": "ami-b73b63a0",
            "instance_ids": null,
            "instance_initiated_shutdown_behavior": null,
            "instance_profile_name": "server",
            "instance_tags": {
                "Name": "host.domain.internal"
            },
            "instance_type": "c3.large",
            "kernel": null,
            "key_name": "host",
            "monitoring": false,
            "network_interfaces": null,
            "placement_group": null,
            "private_ip": null,
            "profile": null,
            "ramdisk": null,
            "region": "us-east-1",
            "security_token": null,
            "source_dest_check": true,
            "spot_launch_group": null,
            "spot_price": null,
            "spot_type": "one-time",
            "spot_wait_timeout": "600",
            "state": "present",
            "tenancy": "default",
            "termination_protection": null,
            "user_data": null,
            "validate_certs": true,
            "volumes": [
                {
                    "delete_on_termination": true,
                    "device_name": "/dev/xvda",
                    "volume_size": 10,
                    "volume_type": "gp2"
                }
            ],
            "vpc_subnet_id": "subnet-********",
            "wait": true,
            "wait_timeout": "300",
            "zone": null
        },
        "module_name": "ec2"
    },
    "msg": "Unable to terminate instance i-aaaaaaaa, error: EC2ResponseError: 400 Bad Request\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Response><Errors><Error><Code>OperationNotPermitted</Code><Message>The instance 'i-aaaaaaaa' may not be terminated. Modify its 'disableApiTermination' instance attribute and try again.</Message></Error></Errors><RequestID>d3a71576-0376-4293-b5ec-9ce9d241fee1</RequestID></Response>"
```



Discussion:
Just to be clear, when I run the above on my inventory of 8 different instances, each with unique hostnames and attributes, I get the same `Unable to terminate instance i-aaaaaaaa, error...` 8 times. The instance it's trying to terminate in every case is the same one running Ansible in a separate VPC/subnet.
The problem is with `count_tag: Name`

Ansible parses this as "any instance with a Name tag"

Try this instead

     count_tag:
        Name: "{{ server_name }}"
     instance_tags:
        Name: "{{ server_name }}"
I can confirm this bug. It seems that if count_tag is set to anything that either returns nil or isn't a tag it will go ahead and terminate unrelated instances. Title should be changed to will terminate instances until it reaches an instance with termination protection turned on.
It looks like the problem starts at the [`enforce_count`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L925) method where [`find_running_instances_by_count_tag`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L936) returns the entire list of instances it can get which in most cases will be all of the instances available to a user. This is because [`get_reservations`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L642) will return every instance a user has access to if `count_tag` isn't a string. So if someone incorrectly sets count_tag as an int. All the checks are passed in [`get_reservations`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L668-L692) until we get to [`if state`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L694). In this case we are looking for all instances that are running in whatever AZ was set. So our filter essentially looks like {'instance-state-name': 'running'}, {'availability-zone', 'us-east-1 or whatever az here'}. This list gets passed back to enforce_count and it goes on a terminate killing spree on all instances that are running and will only be stopped when it hits an instance with termination protection. If a user doesn't have termination protection, they will lose all of their instances.

So I guess there are a couple of ways to fix it but I think the easiest would be handling the case where count_tag was set to an int, doing a type conversion and then checking the final filter tag-key/tag and comparing what was passed via tags.
Just updated with the proposed fix. If anyone has any questions let me know.
cc @lwade @tgerla
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Just to be clear, when I run the above on my inventory of 8 different instances, each with unique hostnames and attributes, I get the same `Unable to terminate instance i-aaaaaaaa, error...` 8 times. The instance it's trying to terminate in every case is the same one running Ansible in a separate VPC/subnet.
The problem is with `count_tag: Name`

Ansible parses this as "any instance with a Name tag"

Try this instead

     count_tag:
        Name: "{{ server_name }}"
     instance_tags:
        Name: "{{ server_name }}"
I can confirm this bug. It seems that if count_tag is set to anything that either returns nil or isn't a tag it will go ahead and terminate unrelated instances. Title should be changed to will terminate instances until it reaches an instance with termination protection turned on.
It looks like the problem starts at the [`enforce_count`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L925) method where [`find_running_instances_by_count_tag`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L936) returns the entire list of instances it can get which in most cases will be all of the instances available to a user. This is because [`get_reservations`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L642) will return every instance a user has access to if `count_tag` isn't a string. So if someone incorrectly sets count_tag as an int. All the checks are passed in [`get_reservations`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L668-L692) until we get to [`if state`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2.py#L694). In this case we are looking for all instances that are running in whatever AZ was set. So our filter essentially looks like {'instance-state-name': 'running'}, {'availability-zone', 'us-east-1 or whatever az here'}. This list gets passed back to enforce_count and it goes on a terminate killing spree on all instances that are running and will only be stopped when it hits an instance with termination protection. If a user doesn't have termination protection, they will lose all of their instances.

So I guess there are a couple of ways to fix it but I think the easiest would be handling the case where count_tag was set to an int, doing a type conversion and then checking the final filter tag-key/tag and comparing what was passed via tags.
Just updated with the proposed fix. If anyone has any questions let me know.
cc @lwade @tgerla
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2.py b/lib/ansible/modules/cloud/amazon/ec2.py
--- a/lib/ansible/modules/cloud/amazon/ec2.py
+++ b/lib/ansible/modules/cloud/amazon/ec2.py
@@ -636,10 +636,10 @@
     HAS_BOTO = False
 
 
-def find_running_instances_by_count_tag(module, ec2, count_tag, zone=None):
+def find_running_instances_by_count_tag(module, ec2, vpc, count_tag, zone=None):
 
     # get reservations for instances that match tag(s) and are running
-    reservations = get_reservations(module, ec2, tags=count_tag, state="running", zone=zone)
+    reservations = get_reservations(module, ec2, vpc, tags=count_tag, state="running", zone=zone)
 
     instances = []
     for res in reservations:
@@ -660,11 +660,20 @@ def _set_none_to_blank(dictionary):
     return result
 
 
-def get_reservations(module, ec2, tags=None, state=None, zone=None):
-
+def get_reservations(module, ec2, vpc, tags=None, state=None, zone=None):
     # TODO: filters do not work with tags that have underscores
     filters = dict()
 
+    vpc_subnet_id = module.params.get('vpc_subnet_id')
+    vpc_id = None
+    if vpc_subnet_id:
+        filters.update({"subnet-id": vpc_subnet_id})
+        if vpc:
+            vpc_id = vpc.get_all_subnets(subnet_ids=[vpc_subnet_id])[0].vpc_id
+
+    if vpc_id:
+        filters.update({"vpc-id": vpc_id})
+
     if tags is not None:
 
         if isinstance(tags, str):
@@ -933,7 +942,7 @@ def enforce_count(module, ec2, vpc):
     if exact_count and count_tag is None:
         module.fail_json(msg="you must use the 'count_tag' option with exact_count")
 
-    reservations, instances = find_running_instances_by_count_tag(module, ec2, count_tag, zone)
+    reservations, instances = find_running_instances_by_count_tag(module, ec2, vpc, count_tag, zone)
 
     changed = None
     checkmode = False


Test patch: