Repository: ansible/ansible

Issue:
Unstable integration test ec2_vpc_net
##### ISSUE TYPE

Bug Report

##### COMPONENT NAME

ec2_vpc_net integration test

##### ANSIBLE VERSION

2.5

##### CONFIGURATION

N/A

##### OS / ENVIRONMENT

Shippable

##### SUMMARY

Unstable integration test ec2_vpc_net:

https://app.shippable.com/github/ansible/ansible/runs/54395/67/tests

##### STEPS TO REPRODUCE

Run test on Shippable.

##### EXPECTED RESULTS

Tests pass.

##### ACTUAL RESULTS

Tests are unstable or fail.

> test/integration/targets/ec2_vpc_net/tasks/main.yml:204 / [localhost] amazon: ec2_vpc_net : create a DHCP option set to use in next test aws_access_key={{ aws_access_key }}, aws_secret_key={{ aws_secret_key }}, security_token={{ security_token }}, region={{ aws_region }}, dns_servers=['4.4.4.4', '8.8.8.8'], tags={'Name': '{{ resource_prefix }}'}

> error: <Response><Errors><Error><Code>InvalidDhcpOptionsID.NotFound</Code><Message>The dhcpOptions ID 'dopt-6cf05914' does not exist</Message></Error></Errors><RequestID>d602f51a-66b5-445f-9b5b-38ffdbccda29</RequestID></Response>

```
Traceback (most recent call last):
  File "/tmp/ansible_oqa3j78q/ansible_module_ec2_vpc_dhcp_option.py", line 233, in ensure_tags
    vpc_conn.create_tags(resource_id, to_add, dry_run=check_mode)
  File "/usr/local/lib/python3.6/dist-packages/boto/ec2/connection.py", line 4219, in create_tags
    return self.get_status('CreateTags', params, verb='POST')
  File "/usr/local/lib/python3.6/dist-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidDhcpOptionsID.NotFound</Code><Message>The dhcpOptions ID 'dopt-6cf05914' does not exist</Message></Error></Errors><RequestID>d602f51a-66b5-445f-9b5b-38ffdbccda29</RequestID></Response>
```


Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/ec2_vpc_net.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2_vpc_net.py)
* [test/integration/targets/ec2_vpc_net/aliases](https://github.com/ansible/ansible/blob/devel/test/integration/targets/ec2_vpc_net/aliases)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @defionscode @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Here's another test failure:

https://app.shippable.com/github/ansible/ansible/runs/54359/66/tests

> test/integration/targets/ec2_vpc_net/tasks/main.yml:63 / [localhost] amazon: ec2_vpc_net : create a VPC aws_secret_key={{ aws_secret_key }}, aws_access_key={{ aws_access_key }}, name={{ resource_prefix }}, security_token={{ security_token }}, state=present, cidr_block=20.0.0.0/24, region={{ aws_region }}

> error: ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-4f1fed34' does not exist

```
Traceback (most recent call last):
  File "/tmp/ansible_HMtsGa/ansible_module_ec2_vpc_net.py", line 227, in update_vpc_tags
    connection.create_tags(Resources=[vpc_id], Tags=tags)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-4f1fed34' does not exist
```

> test/integration/targets/ec2_vpc_net/tasks/main.yml:63 / [localhost] amazon: ec2_vpc_net : create a VPC aws_secret_key={{ aws_secret_key }}, aws_access_key={{ aws_access_key }}, name={{ resource_prefix }}, security_token={{ security_token }}, state=present, cidr_block=20.0.0.0/24, region={{ aws_region }}

> error: ClientError: An error occurred (InvalidVpcID.NotFound) when calling the DescribeVpcs operation: The vpc ID 'vpc-ce10e2b5' does not exist

```
Traceback (most recent call last):
  File "/tmp/ansible_ukwhoU/ansible_module_ec2_vpc_net.py", line 199, in get_vpc
    vpc_obj = connection.describe_vpcs(VpcIds=[vpc_id])['Vpcs'][0]
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidVpcID.NotFound) when calling the DescribeVpcs operation: The vpc ID 'vpc-ce10e2b5' does not exist
```
Here's another occurrence: https://app.shippable.com/github/ansible/ansible/runs/54507/66/tests
This test continues to be unstable:

https://app.shippable.com/github/ansible/ansible/runs/57503/67/tests
Here's another failure: https://app.shippable.com/github/ansible/ansible/runs/57633/67/tests
Additional failure: https://app.shippable.com/github/ansible/ansible/runs/57939/67/tests
Another failure: https://app.shippable.com/github/ansible/ansible/runs/57950/66/tests
Another failure: https://app.shippable.com/github/ansible/ansible/runs/60530/66/tests

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2_vpc_dhcp_option.py b/lib/ansible/modules/cloud/amazon/ec2_vpc_dhcp_option.py
--- a/lib/ansible/modules/cloud/amazon/ec2_vpc_dhcp_option.py
+++ b/lib/ansible/modules/cloud/amazon/ec2_vpc_dhcp_option.py
@@ -201,6 +201,18 @@ def get_resource_tags(vpc_conn, resource_id):
     return dict((t.name, t.value) for t in vpc_conn.get_all_tags(filters={'resource-id': resource_id}))
 
 
+def retry_not_found(to_call, *args, **kwargs):
+    start_time = time()
+    while time() < start_time + 300:
+        try:
+            return to_call(*args, **kwargs)
+        except EC2ResponseError as e:
+            if e.error_code == 'InvalidDhcpOptionID.NotFound':
+                sleep(3)
+                continue
+            raise e
+
+
 def ensure_tags(module, vpc_conn, resource_id, tags, add_only, check_mode):
     try:
         cur_tags = get_resource_tags(vpc_conn, resource_id)
@@ -209,11 +221,11 @@ def ensure_tags(module, vpc_conn, resource_id, tags, add_only, check_mode):
 
         to_delete = dict((k, cur_tags[k]) for k in cur_tags if k not in tags)
         if to_delete and not add_only:
-            vpc_conn.delete_tags(resource_id, to_delete, dry_run=check_mode)
+            retry_not_found(vpc_conn.delete_tags, resource_id, to_delete, dry_run=check_mode)
 
         to_add = dict((k, tags[k]) for k in tags if k not in cur_tags)
         if to_add:
-            vpc_conn.create_tags(resource_id, to_add, dry_run=check_mode)
+            retry_not_found(vpc_conn.create_tags, resource_id, to_add, dry_run=check_mode)
 
         latest_tags = get_resource_tags(vpc_conn, resource_id)
         return {'changed': True, 'tags': latest_tags}
@@ -363,16 +375,10 @@ def main():
             # wait for dhcp option to be accessible
             found_dhcp_opt = False
             start_time = time()
-            while time() < start_time + 300:
-                try:
-                    found_dhcp_opt = connection.get_all_dhcp_options(dhcp_options_ids=[dhcp_option.id])
-                except EC2ResponseError as e:
-                    if e.error_code == 'InvalidDhcpOptionID.NotFound':
-                        sleep(3)
-                    else:
-                        module.fail_json(msg="Failed to describe DHCP options", exception=traceback.format_exc)
-                else:
-                    break
+            try:
+                found_dhcp_opt = retry_not_found(connection.get_all_dhcp_options, dhcp_options_ids=[dhcp_option.id])
+            except EC2ResponseError as e:
+                module.fail_json(msg="Failed to describe DHCP options", exception=traceback.format_exc)
             if not found_dhcp_opt:
                 module.fail_json(msg="Failed to wait for {0} to be available.".format(dhcp_option.id))
 


Test patch: