Repository: ansible/ansible

Issue:
Unstable integration test ec2_vpc_subnet
##### ISSUE TYPE

Bug Report

##### COMPONENT NAME

ec2_vpc_subnet integration test

##### ANSIBLE VERSION

2.5

##### CONFIGURATION

N/A

##### OS / ENVIRONMENT

Shippable

##### SUMMARY

Unstable integration test ec2_vpc_subnet:

https://app.shippable.com/github/ansible/ansible/runs/54187/55/tests

##### STEPS TO REPRODUCE

Run tests on Shippable.

##### EXPECTED RESULTS

Tests pass.

##### ACTUAL RESULTS

Tests are unstable or fail.

> test/integration/targets/ec2_vpc_subnet/tasks/main.yml:350 / [localhost] amazon: ec2_vpc_subnet : tidy up subnet aws_secret_key={{ aws_secret_key }}, aws_access_key={{ aws_access_key }}, security_token={{ security_token }}, state=absent, vpc_id={{ vpc_result.vpc.id }}, region={{ ec2_region }}, cidr=10.232.232.128/28

> failure: The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'vpc' The error appears to have been in '/root/ansible/test/integration/targets/ec2_vpc_subnet/tasks/main.yml': line 350, column 7, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: tidy up subnet ^ here

```
{
"failed": true, 
"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'vpc'\n\nThe error appears to have been in '/root/ansible/test/integration/targets/ec2_vpc_subnet/tasks/main.yml': line 350, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: tidy up subnet\n      ^ here\n"
}
```

> test/integration/targets/ec2_vpc_subnet/tasks/main.yml:14 / [localhost] amazon: ec2_vpc_subnet : create a VPC aws_secret_key={{ aws_secret_key }}, aws_access_key={{ aws_access_key }}, name={{ resource_prefix }}-vpc, security_token={{ security_token }}, tags={u'Name': u'{{ resource_prefix }}-vpc', u'Description': u'Created by ansible-test'}, region={{ ec2_region }}, state=present, cidr_block=10.232.232.128/26

> error: ClientError: An error occurred (InvalidVpcID.NotFound) when calling the DescribeVpcClassicLink operation: The vpc ID 'vpc-1b2bda60' does not exist

```
Traceback (most recent call last):
  File "/tmp/ansible_dPE6PL/ansible_module_ec2_vpc_net.py", line 203, in get_vpc
    classic_link = connection.describe_vpc_classic_link(VpcIds=[vpc_id])['Vpcs'][0].get('ClassicLinkEnabled')
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidVpcID.NotFound) when calling the DescribeVpcClassicLink operation: The vpc ID 'vpc-1b2bda60' does not exist
```


Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/ec2_vpc_subnet.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/ec2_vpc_subnet.py)
* [test/integration/targets/ec2_vpc_subnet/aliases](https://github.com/ansible/ansible/blob/devel/test/integration/targets/ec2_vpc_subnet/aliases)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @brandond @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
This test continues to be unstable:

https://app.shippable.com/github/ansible/ansible/runs/57503/66/tests

Here's another failure: https://app.shippable.com/github/ansible/ansible/runs/57626/66/tests
Another failure: https://app.shippable.com/github/ansible/ansible/runs/57929/66/tests

However, the failure is different than the one originally reported:

> test/integration/targets/ec2_vpc_subnet/tasks/main.yml:337 / [localhost] amazon: ec2_vpc_subnet : assert subnet ipv6 cidr removed (expected changed=true) that=[u'vpc_remove_ipv6_cidr.changed', u"vpc_remove_ipv6_cidr.subnet.ipv6_cidr_block == ''", u'not vpc_remove_ipv6_cidr.subnet.assign_ipv6_address_on_creation']

> failure: rc=0

```
{
    "assertion": "vpc_remove_ipv6_cidr.subnet.ipv6_cidr_block == ''", 
    "changed": false, 
    "evaluated_to": false
}
```
Another failure: https://app.shippable.com/github/ansible/ansible/runs/57935/66/tests
Partially fixed by: #37354
Still needs ec2_vpc_subnet stabilization
Another failure: https://app.shippable.com/github/ansible/ansible/runs/58876/66/tests
Disabled test in https://github.com/ansible/ansible/commit/af15a46f20b152a80a6d714bd519950d6fd9ab0c
Reenabled test in 873a9ddf8dc37086aa8998c248310a23ae518c9d. 
It looks like these tests are still unstable:

https://app.shippable.com/github/ansible/ansible/runs/60100/67/tests
Another unstable test run: https://app.shippable.com/github/ansible/ansible/runs/61381/67/tests
@mattclay that test is weird. The test shows that the assertion fails but the console log shows the assertion passes:

```
2018-04-14 07:37:44 
2018-04-14 07:37:44 TASK [ec2_vpc_subnet : assert subnet changed] **********************************
2018-04-14 07:37:44 task path: /root/ansible/test/integration/targets/ec2_vpc_subnet/tasks/main.yml:142
2018-04-14 07:37:44 ok: [localhost] => {
2018-04-14 07:37:44     "changed": false,
2018-04-14 07:37:44     "msg": "All assertions passed"
2018-04-14 07:37:44 }
```

The assertion *should* pass, as well, because the results from the preceding task are as expected.

If there is a second run of ec2_vpc_subnet, let me know, but searching for `:142` in the console log only showed one result for this assertion in the build_ci section.
@willthames Yes, the test ran twice. It failed on the first run:

```
2018-04-14 07:37:20 TASK [ec2_vpc_subnet : assert subnet changed] **********************************
2018-04-14 07:37:21 fatal: [localhost]: FAILED! => {
2018-04-14 07:37:21     "assertion": "vpc_subnet_modify.subnet.map_public_ip_on_launch",
2018-04-14 07:37:21     "changed": false,
2018-04-14 07:37:21     "evaluated_to": false
2018-04-14 07:37:21 }
```

Then the auto-retry kicked in:

```
2018-04-14 07:37:25 WARNING: Retrying test target "ec2_vpc_subnet" with maximum verbosity.
```

And it passed on the second run:

```
2018-04-14 07:37:44 TASK [ec2_vpc_subnet : assert subnet changed] **********************************
2018-04-14 07:37:44 task path: /root/ansible/test/integration/targets/ec2_vpc_subnet/tasks/main.yml:142
2018-04-14 07:37:44 ok: [localhost] => {
2018-04-14 07:37:44     "changed": false,
2018-04-14 07:37:44     "msg": "All assertions passed"
2018-04-14 07:37:44 }
```

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/ec2_vpc_subnet.py b/lib/ansible/modules/cloud/amazon/ec2_vpc_subnet.py
--- a/lib/ansible/modules/cloud/amazon/ec2_vpc_subnet.py
+++ b/lib/ansible/modules/cloud/amazon/ec2_vpc_subnet.py
@@ -508,7 +508,7 @@ def ensure_subnet_present(conn, module):
 
 
 def ensure_final_subnet(conn, module, subnet, start_time):
-    for rewait in range(0, 10):
+    for rewait in range(0, 30):
         map_public_correct = False
         assign_ipv6_correct = False
 
@@ -531,7 +531,7 @@ def ensure_final_subnet(conn, module, subnet, start_time):
         if map_public_correct and assign_ipv6_correct:
             break
 
-        time.sleep(3)
+        time.sleep(5)
         subnet = get_matching_subnet(conn, module, module.params['vpc_id'], module.params['cidr'])
 
     return subnet


Test patch: