Repository: ansible/ansible

Issue:
Unstable integration test ec2_vpc_net
##### ISSUE TYPE

Bug Report

##### COMPONENT NAME

ec2_vpc_net integration test

##### ANSIBLE VERSION

2.5

##### CONFIGURATION

N/A

##### OS / ENVIRONMENT

Shippable

##### SUMMARY

Unstable integration test ec2_vpc_net:

https://app.shippable.com/github/ansible/ansible/runs/54395/67/tests

##### STEPS TO REPRODUCE

Run test on Shippable.

##### EXPECTED RESULTS

Tests pass.

##### ACTUAL RESULTS

Tests are unstable or fail.

> test/integration/targets/ec2_vpc_net/tasks/main.yml:204 / [localhost] amazon: ec2_vpc_net : create a DHCP option set to use in next test aws_access_key={{ aws_access_key }}, aws_secret_key={{ aws_secret_key }}, security_token={{ security_token }}, region={{ aws_region }}, dns_servers=['4.4.4.4', '8.8.8.8'], tags={'Name': '{{ resource_prefix }}'}

> error: <Response><Errors><Error><Code>InvalidDhcpOptionsID.NotFound</Code><Message>The dhcpOptions ID 'dopt-6cf05914' does not exist</Message></Error></Errors><RequestID>d602f51a-66b5-445f-9b5b-38ffdbccda29</RequestID></Response>

```
Traceback (most recent call last):
  File "/tmp/ansible_oqa3j78q/ansible_module_ec2_vpc_dhcp_option.py", line 233, in ensure_tags
    vpc_conn.create_tags(resource_id, to_add, dry_run=check_mode)
  File "/usr/local/lib/python3.6/dist-packages/boto/ec2/connection.py", line 4219, in create_tags
    return self.get_status('CreateTags', params, verb='POST')
  File "/usr/local/lib/python3.6/dist-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidDhcpOptionsID.NotFound</Code><Message>The dhcpOptions ID 'dopt-6cf05914' does not exist</Message></Error></Errors><RequestID>d602f51a-66b5-445f-9b5b-38ffdbccda29</RequestID></Response>
```


Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/ec2_vpc_net.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2_vpc_net.py)
* [test/integration/targets/ec2_vpc_net/aliases](https://github.com/ansible/ansible/blob/devel/test/integration/targets/ec2_vpc_net/aliases)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @defionscode @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Here's another test failure:

https://app.shippable.com/github/ansible/ansible/runs/54359/66/tests

> test/integration/targets/ec2_vpc_net/tasks/main.yml:63 / [localhost] amazon: ec2_vpc_net : create a VPC aws_secret_key={{ aws_secret_key }}, aws_access_key={{ aws_access_key }}, name={{ resource_prefix }}, security_token={{ security_token }}, state=present, cidr_block=20.0.0.0/24, region={{ aws_region }}

> error: ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-4f1fed34' does not exist

```
Traceback (most recent call last):
  File "/tmp/ansible_HMtsGa/ansible_module_ec2_vpc_net.py", line 227, in update_vpc_tags
    connection.create_tags(Resources=[vpc_id], Tags=tags)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-4f1fed34' does not exist
```

> test/integration/targets/ec2_vpc_net/tasks/main.yml:63 / [localhost] amazon: ec2_vpc_net : create a VPC aws_secret_key={{ aws_secret_key }}, aws_access_key={{ aws_access_key }}, name={{ resource_prefix }}, security_token={{ security_token }}, state=present, cidr_block=20.0.0.0/24, region={{ aws_region }}

> error: ClientError: An error occurred (InvalidVpcID.NotFound) when calling the DescribeVpcs operation: The vpc ID 'vpc-ce10e2b5' does not exist

```
Traceback (most recent call last):
  File "/tmp/ansible_ukwhoU/ansible_module_ec2_vpc_net.py", line 199, in get_vpc
    vpc_obj = connection.describe_vpcs(VpcIds=[vpc_id])['Vpcs'][0]
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidVpcID.NotFound) when calling the DescribeVpcs operation: The vpc ID 'vpc-ce10e2b5' does not exist
```
Here's another occurrence: https://app.shippable.com/github/ansible/ansible/runs/54507/66/tests
This test continues to be unstable:

https://app.shippable.com/github/ansible/ansible/runs/57503/67/tests
Here's another failure: https://app.shippable.com/github/ansible/ansible/runs/57633/67/tests
Additional failure: https://app.shippable.com/github/ansible/ansible/runs/57939/67/tests
Another failure: https://app.shippable.com/github/ansible/ansible/runs/57950/66/tests
Another failure: https://app.shippable.com/github/ansible/ansible/runs/60530/66/tests
Failing again: https://app.shippable.com/github/ansible/ansible/runs/62516/67/tests

Resolving patch:
diff --git a/lib/ansible/module_utils/aws/core.py b/lib/ansible/module_utils/aws/core.py
--- a/lib/ansible/module_utils/aws/core.py
+++ b/lib/ansible/module_utils/aws/core.py
@@ -177,6 +177,11 @@ def fail_json_aws(self, exception, msg=None):
 
 
 class _RetryingBotoClientWrapper(object):
+    __never_wait = (
+        'get_paginator', 'can_paginate',
+        'get_waiter', 'generate_presigned_url',
+    )
+
     def __init__(self, client, retry):
         self.client = client
         self.retry = retry
@@ -194,7 +199,9 @@ def deciding_wrapper(aws_retry=False, *args, **kwargs):
 
     def __getattr__(self, name):
         unwrapped = getattr(self.client, name)
-        if callable(unwrapped):
+        if name in self.__never_wait:
+            return unwrapped
+        elif callable(unwrapped):
             wrapped = self._create_optional_retry_wrapper_function(unwrapped)
             setattr(self, name, wrapped)
             return wrapped
diff --git a/lib/ansible/modules/cloud/amazon/ec2_vpc_net.py b/lib/ansible/modules/cloud/amazon/ec2_vpc_net.py
--- a/lib/ansible/modules/cloud/amazon/ec2_vpc_net.py
+++ b/lib/ansible/modules/cloud/amazon/ec2_vpc_net.py
@@ -166,8 +166,8 @@
 
 from time import sleep, time
 from ansible.module_utils.aws.core import AnsibleAWSModule
-from ansible.module_utils.ec2 import (boto3_conn, get_aws_connection_info, ec2_argument_spec, camel_dict_to_snake_dict,
-                                      ansible_dict_to_boto3_tag_list, boto3_tag_list_to_ansible_dict, AWSRetry)
+from ansible.module_utils.ec2 import (AWSRetry, camel_dict_to_snake_dict,
+                                      ansible_dict_to_boto3_tag_list, boto3_tag_list_to_ansible_dict)
 from ansible.module_utils.six import string_types
 
 
@@ -312,8 +312,7 @@ def wait_for_vpc_attribute(connection, module, vpc_id, attribute, expected_value
 
 
 def main():
-    argument_spec = ec2_argument_spec()
-    argument_spec.update(dict(
+    argument_spec = dict(
         name=dict(required=True),
         cidr_block=dict(type='list', required=True),
         tenancy=dict(choices=['default', 'dedicated'], default='default'),
@@ -325,7 +324,6 @@ def main():
         multi_ok=dict(type='bool', default=False),
         purge_cidrs=dict(type='bool', default=False),
     )
-    )
 
     module = AnsibleAWSModule(
         argument_spec=argument_spec,
@@ -345,8 +343,12 @@ def main():
 
     changed = False
 
-    region, ec2_url, aws_connect_params = get_aws_connection_info(module, boto3=True)
-    connection = boto3_conn(module, conn_type='client', resource='ec2', region=region, endpoint=ec2_url, **aws_connect_params)
+    connection = module.client(
+        'ec2',
+        retry_decorator=AWSRetry.jittered_backoff(
+            retries=8, delay=3, catch_extra_error_codes=['InvalidVpcID.NotFound']
+        )
+    )
 
     if dns_hostnames and not dns_support:
         module.fail_json(msg='In order to enable DNS Hostnames you must also enable DNS support')
@@ -396,8 +398,8 @@ def main():
             except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
                 module.fail_json_aws(e, msg="Failed to update tags")
 
-        current_dns_enabled = connection.describe_vpc_attribute(Attribute='enableDnsSupport', VpcId=vpc_id)['EnableDnsSupport']['Value']
-        current_dns_hostnames = connection.describe_vpc_attribute(Attribute='enableDnsHostnames', VpcId=vpc_id)['EnableDnsHostnames']['Value']
+        current_dns_enabled = connection.describe_vpc_attribute(Attribute='enableDnsSupport', VpcId=vpc_id, aws_retry=True)['EnableDnsSupport']['Value']
+        current_dns_hostnames = connection.describe_vpc_attribute(Attribute='enableDnsHostnames', VpcId=vpc_id, aws_retry=True)['EnableDnsHostnames']['Value']
         if current_dns_enabled != dns_support:
             changed = True
             if not module.check_mode:


Test patch: