Repository: ansible/ansible

Issue:
ec2.py now fails with "subnet ID does not exist"
##### ISSUE TYPE
 - Bug Report


##### COMPONENT NAME
module/cloud/amazon/ec2.py

##### ANSIBLE VERSION
```
ansible 2.4.0 (devel cf30b162a9) last updated 2017/06/14 11:24:28 (GMT +1300)
  config file = /Users/jmorgan/git/sovereign/ansible/ansible.cfg
  configured module search path = [u'/Users/jmorgan/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/jmorgan/tmp/ansible/lib/ansible
  executable location = /Users/jmorgan/tmp/ansible/bin/ansible
  python version = 2.7.11 (default, Jan 22 2016, 08:29:18) [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]
```

##### CONFIGURATION
<!---
```
ssh_args = -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ForwardAgent=yes
host_key_checking = False
nocows = 1
hostfile = inventory/ec2.py
remote_tmp = /tmp/ansible-$USER/tmp
gathering = smart
pipelining = True
log_path = /tmp/ansible.log
#callback_plugins = library/callback
#filter_plugins = /usr/share/ansible_plugins/filter_plugins:library/plugins
```
-->

##### OS / ENVIRONMENT
OSX 10.11.6

##### SUMMARY
When running the ec2 module to deploy an AWS EC2 instance, it now results in a "Subnet ID does not exist" error. It worked a couple days ago and I just pulled down the latest devel commits from the repository. Rolling back to the previous commit for ec2.py makes the playbook work again. 

##### STEPS TO REPRODUCE
Running the yaml below in a playbook with var files that provide the values. 
the vpc_subnet_id is being provided with a correct existing subnet for a VPC

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: AWS | Create EC2 Nodes
  local_action:
    module: ec2
    aws_access_key: "{{ sts_aws_access_key | default(omit) }}"
    aws_secret_key: "{{ sts_aws_secret_key | default(omit) }}"
    security_token: "{{ sts_security_token | default(omit) }}"
    group_id: "{{sg.group_id}}"
    instance_type: "{{instance_type}}"
    image: "{{image}}"
    wait: "yes"
    vpc_subnet_id: "{{subnet_info.subnets[0].id}}"
    assign_public_ip: "{{assign_public_ip}}"
    region: "{{region}}"
    key_name: "{{key_name}}"
    user_data: "{{bash_script | default(omit)}}"
    volumes: "{{volumes | default(omit)}}"
    instance_profile_name: "{{iam_profile_name | default(omit)}}"
    private_ip: "{{ipaddress | default(omit) }}"
    termination_protection: "{{termination_protection | default(omit)}}"
    monitoring: "yes"
    instance_tags:
      Name: "{{org_name}}-{{env}}-{{hostname}}"
      Org: "{{org_name}}"
      Environment: "{{env}}"
      Vpc: "{{vpc}}"
      Project: "{{tag_project}}"
      Costcenter: "{{tag_costcenter}}"
      Region: "{{region}}"
      Service: "{{service}}"
      AZ: "{{az}}"
    exact_count: "{{how_many}}"
    count_tag:
      Name: "{{org_name}}-{{env}}-{{hostname}}"
      Org: "{{org_name}}"
      Environment: "{{env}}"
      #Infrastructure: "{{vpc}}"
      Project: "{{tag_project}}"
      #Costcenter: "{{tag_costcenter}}"
      #Region: "{{region}}"
      Service: "{{service}}"
      AZ: "{{az}}"
  #with_items: "{{subnet}}"
  register: ec2
  tags:
    - build
    - configure
    - dns
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
I expected it to complete the creation of an EC2 instance

##### ACTUAL RESULTS
It returned the result below

```
Using module_utils file /Users/jmorgan/tmp/ansible/lib/ansible/module_utils/basic.py
Using module_utils file /Users/jmorgan/tmp/ansible/lib/ansible/module_utils/ec2.py
Using module_utils file /Users/jmorgan/tmp/ansible/lib/ansible/module_utils/six/__init__.py
Using module_utils file /Users/jmorgan/tmp/ansible/lib/ansible/module_utils/_text.py
Using module_utils file /Users/jmorgan/tmp/ansible/lib/ansible/module_utils/pycompat24.py
Using module_utils file /Users/jmorgan/tmp/ansible/lib/ansible/module_utils/cloud.py
Using module_utils file /Users/jmorgan/tmp/ansible/lib/ansible/module_utils/six/_six.py
Using module file /Users/jmorgan/tmp/ansible/lib/ansible/modules/cloud/amazon/ec2.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: jmorgan
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-$USER/tmp/ansible-tmp-1497405139.06-173735515787978 `" && echo ansible-tmp-1497405139.06-173735515787978="` echo /tmp/ansible-$USER/tmp/ansible-tmp-1497405139.06-173735515787978 `" ) && sleep 0'
<localhost> PUT /var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/tmp2mUYrZ TO /tmp/ansible-jmorgan/tmp/ansible-tmp-1497405139.06-173735515787978/ec2.py
<localhost> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-jmorgan/tmp/ansible-tmp-1497405139.06-173735515787978/ /tmp/ansible-jmorgan/tmp/ansible-tmp-1497405139.06-173735515787978/ec2.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/local/opt/python/bin/python2.7 /tmp/ansible-jmorgan/tmp/ansible-tmp-1497405139.06-173735515787978/ec2.py; rm -rf "/tmp/ansible-jmorgan/tmp/ansible-tmp-1497405139.06-173735515787978/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/ansible_Q79IEm/ansible_module_ec2.py", line 1675, in <module>
    main()
  File "/var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/ansible_Q79IEm/ansible_module_ec2.py", line 1669, in main
    (tagged_instances, instance_dict_array, new_instance_ids, changed) = enforce_count(module, ec2, vpc)
  File "/var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/ansible_Q79IEm/ansible_module_ec2.py", line 949, in enforce_count
    = create_instances(module, ec2, vpc, override_count=to_create)
  File "/var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/ansible_Q79IEm/ansible_module_ec2.py", line 1034, in create_instances
    vpc_id = vpc.get_all_subnets(subnet_ids=[vpc_subnet_id])[0].vpc_id
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/vpc/__init__.py", line 1153, in get_all_subnets
    return self.get_list('DescribeSubnets', params, [('item', Subnet)])
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/connection.py", line 1186, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidSubnetID.NotFound</Code><Message>The subnet ID 'subnet-42c96f34' does not exist</Message></Error></Errors><RequestID>2e1f90ed-9c0d-4490-a4e8-e2005985a3b3</RequestID></Response>

fatal: [localhost -> localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/ansible_Q79IEm/ansible_module_ec2.py\", line 1675, in <module>\n    main()\n  File \"/var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/ansible_Q79IEm/ansible_module_ec2.py\", line 1669, in main\n    (tagged_instances, instance_dict_array, new_instance_ids, changed) = enforce_count(module, ec2, vpc)\n  File \"/var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/ansible_Q79IEm/ansible_module_ec2.py\", line 949, in enforce_count\n    = create_instances(module, ec2, vpc, override_count=to_create)\n  File \"/var/folders/yt/1w3x85v95qd3kt50624tzgww0000gn/T/ansible_Q79IEm/ansible_module_ec2.py\", line 1034, in create_instances\n    vpc_id = vpc.get_all_subnets(subnet_ids=[vpc_subnet_id])[0].vpc_id\n  File \"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/vpc/__init__.py\", line 1153, in get_all_subnets\n    return self.get_list('DescribeSubnets', params, [('item', Subnet)])\n  File \"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/connection.py\", line 1186, in get_list\n    raise self.ResponseError(response.status, response.reason, body)\nboto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Response><Errors><Error><Code>InvalidSubnetID.NotFound</Code><Message>The subnet ID 'subnet-42c96f34' does not exist</Message></Error></Errors><RequestID>2e1f90ed-9c0d-4490-a4e8-e2005985a3b3</RequestID></Response>\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 0
}
	to retry, use: --limit @/Users/jmorgan/git/sovereign/ansible/deploy.retry

PLAY RECAP **********************************************************************************************************************************************************************************************
localhost                  : ok=6    changed=1    unreachable=0    failed=1
```



Discussion:
@darknessnz Thanks for filing the bug report! Sorry this is breaking for you. I'm having a hard time generating the error with your reproducer. Can you run this and see if the subnet is correctly found? (I'm wondering if could it be in another region)
```
---
- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: get facts
      ec2_vpc_subnet_facts:
        aws_access_key: "{{ sts_aws_access_key | default(omit) }}"
        aws_secret_key: "{{ sts_aws_secret_key | default(omit) }}"
        security_token: "{{ sts_security_token | default(omit) }}"  
        region: "{{ region }}"
        filters:
          vpc-id: "{{ vpc }}"
          subnet-id: "{{ subnet }}"
```
Also, if you're feeling inclined to test something, I'm wondering if you playbook worked prior to #24470 being merged, which allowed VPCConnection to be created with ec2_url as well. If you can confirm/deny that it would help figure out what's going on too.
Hi, thanks for the response. The example you posted won't work for my environment (as subnet is a name not id) but I'm actually doing something similar right before the ec2 modules call.

```
    - name: get facts
      ec2_vpc_subnet_facts:
        aws_access_key: "{{ sts_aws_access_key | default(omit) }}"
        aws_secret_key: "{{ sts_aws_secret_key | default(omit) }}"
        security_token: "{{ sts_security_token | default(omit) }}"
        region: "{{ region }}"
        filters:
          "tag:Name": "{{org_name}}-{{vpc}}-{{subnet}}-{{az}}"
      register: subnet_info
```

This definitely returns the subnet I need to deploy the ec2 instance into. With the register value providing the actual subnet id values via subnet_info.subnets[0].id . They are definitely all in the same region. I'll work on testing that other pull request section. Though this code has been working since last Nov when I started on it. Only just started having problems when I pulled the latest devel branch yesterday.

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2.py b/lib/ansible/modules/cloud/amazon/ec2.py
--- a/lib/ansible/modules/cloud/amazon/ec2.py
+++ b/lib/ansible/modules/cloud/amazon/ec2.py
@@ -1642,12 +1642,15 @@ def main():
         module.fail_json(msg='boto required for this module')
 
     try:
-        _, ec2_url, aws_connect_kwargs = get_aws_connection_info(module)
+        region, ec2_url, aws_connect_kwargs = get_aws_connection_info(module)
         if module.params.get('region') or not module.params.get('ec2_url'):
             ec2 = ec2_connect(module)
         elif module.params.get('ec2_url'):
             ec2 = connect_ec2_endpoint(ec2_url, **aws_connect_kwargs)
 
+        if 'region' not in aws_connect_kwargs:
+            aws_connect_kwargs['region'] = ec2.region
+
         vpc = connect_vpc(**aws_connect_kwargs)
     except boto.exception.NoAuthHandlerFound as e:
         module.fail_json(msg="Failed to get connection: %s" % e.message, exception=traceback.format_exc())


Test patch: