Repository: ansible/ansible-modules-core

Issue:
cloudformation module on Ansible 2.2.0 throws "PhysicalResourceId"  error intermittently.
<!--- Verify first that your issue/request is not already reported in GitHub -->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
cloudformation

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.2.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Ubuntu 14.04
##### SUMMARY
<!--- Explain the problem briefly -->
cloudformation module on Ansible 2.2.0 throws "PhysicalResourceId"  error intermittently.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->
Push a stack with Ansible cloudformation module on Ansible 2.2.0

<!--- Paste example playbooks or commands between quotes below -->
```

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'PhysicalResourceId'
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_Kx3c0c/ansible_module_cloudformation.py\", line 483, in <module>\n    main()\n  File \"/tmp/ansible_Kx3c0c/ansible_module_cloudformation.py\", line 450, in main\n    \"physical_resource_id\": res['PhysicalResourceId'],\nKeyError: 'PhysicalResourceId'\n", "module_stdout": "", "msg": "MODULE FAILURE"}
	to retry, use: --limit 
```



Discussion:
<!-- @alanroche: Thanks for filing this bug report! -->

@tedder, @ryansb, ping. This issue is waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_bug_new --->

What's your boto version where this is running? For what it's worth, 2.3 will have a completely rewritten module using boto3.

Now that you mention it, we did upgrade boto to a newer version recently...

## $ pip show boto

Name: boto
Version: 2.43.0
Location: /usr/local/lib/python2.7/dist-packages
Requires:
$

FYI We also have boto3 installed side by side

## $ pip show boto3

Name: boto3
Version: 1.4.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: jmespath, botocore, s3transfer
$

and botocore....

## $ pip show botocore

Name: botocore
Version: 1.4.26
Location: /usr/local/lib/python2.7/dist-packages
Requires: docutils, jmespath, python-dateutil
$

If something has changed in boto2. - we can pin the boto2 version as a workaround until fixed

Can you give a self-contained example that triggers this? 

Unfortunately I cannot really, as it is intermittent. Sometimes the same code works other times not. 
What is happening is pretty simple, - it is just pushing a new cloudformation stack.
I have however managed to repro this with verbose trace enabled if that is of any help.

ASK [cf-api-gateway : Provision Stack] ****************************
task path: /var/lib/jenkins/workspace/[REDACTED]/aws/ansible/roles/cf-api-gateway/tasks/main.yml:14
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/core/cloud/amazon/cloudformation.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: jenkins
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "`echo $HOME/.ansible/tmp/ansible-tmp-1478264178.82-121111996869655`" && echo ansible-tmp-1478264178.82-121111996869655="`echo $HOME/.ansible/tmp/ansible-tmp-1478264178.82-121111996869655`" ) && sleep 0'
<localhost> PUT /tmp/tmp4Ywzvm TO /var/lib/jenkins/.ansible/tmp/ansible-tmp-1478264178.82-121111996869655/cloudformation.py
<localhost> EXEC /bin/sh -c 'chmod u+x /var/lib/jenkins/.ansible/tmp/ansible-tmp-1478264178.82-121111996869655/ /var/lib/jenkins/.ansible/tmp/ansible-tmp-1478264178.82-121111996869655/cloudformation.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /var/lib/jenkins/.ansible/tmp/ansible-tmp-1478264178.82-121111996869655/cloudformation.py; rm -rf "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1478264178.82-121111996869655/" > /dev/null 2>&1 && sleep 0'
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_2ypQgd/ansible_module_cloudformation.py", line 483, in <module>
    main()
  File "/tmp/ansible_2ypQgd/ansible_module_cloudformation.py", line 450, in main
    "physical_resource_id": res['PhysicalResourceId'],
KeyError: 'PhysicalResourceId'
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_name": "cloudformation"
    }, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_2ypQgd/ansible_module_cloudformation.py\", line 483, in <module>\n    main()\n  File \"/tmp/ansible_2ypQgd/ansible_module_cloudformation.py\", line 450, in main\n    \"physical_resource_id\": res['PhysicalResourceId'],\nKeyError: 'PhysicalResourceId'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE"
}
    to retry, use: --limit @/var/lib/jenkins/workspace/[REDACTED]/aws/ansible/push.retry

I can confirm now that this exhibits on boto 2.41.0 also, which we had been using previously and did not observe errors on this version of boto prior to updating to Ansible 2.2. 

PS. We need to use Ansible 2.2 as it seems to fix some errors with SSHing through AWS bastion servers:
https://github.com/ansible/ansible/issues/17349

Hm, I wasn't able to repro this yesterday. Going to try a different way today, but if you're able to provide any info about your playbook that'd help. Does it have a particular resource type? Do you use outputs? 

OK,

I have some useful info now (I think) :)

The stack is failing to push, It was not obvious because it is a test stack that gets dropped after tests get run. Sorry about that, - but because the stack was getting dropped, it appeared on the surface as an intermittent error.  When I go into the CloudFormation console and query back on "Deleted Stacks" I can see clearly that the offending stack is ERRORED (In our case simply due to a CloudWatch logs limit being reached, but that is besides the point).

So, - I am guessing/thinking perhaps that: 
1/ This is a new stack being pushed (ie. NOT an update to existing stack)
2/ The pusgh errored before the physical_resource_id was assigned in Cloudformation, - but maybe the ansible code assumes somewhere that a physical resource ID will come back, - but there is none due to the stack push being errored.

Hope this helps...

And YES - we do have output variables on this stack also, if that matters. The resource being pushed in this case was an AWS API Gateway instance.

Okay. It does seem like an edge case. I'm curious if you can repro in 2.3, as everything is different. Until then, marking this **wontfix**.

I can replicate this error with a stack to create an EC2 Instance that has DisableApiTermination: true and a role that does not have permission to complete the stack.

The stack goes into ROLLBACK_FAILED due to not being able to terminate the instance, and KeyError: 'PhysicalResourceId' occurs.

Ryan, please attempt to repro on the 2.3/devel version of the module.

I reproduced on latest 2.3, here's how I did it:
1. Create an IAM role that has read permissions, and ability to create an S3 bucket (policy below)
2. Create a template that includes an S3 bucket and another resource that the IAM role has no permissions for (I used SNS topic)
3. Use the module to create the stack, using the IAM role specified above

Template:

```
{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters": {},
    "Resources": {
        "Bukkit": {
            "Type": "AWS::S3::Bucket",
            "Properties": {}
        },
        "other": {
            "Type": "AWS::SNS::Topic",
            "Properties": {}
        }
    }
}
```

Policy: 

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1478712625000",
            "Effect": "Allow",
            "Action": ["s3:CreateBucket"],
            "Resource": "*"
        }
    ]
}
```

Playbook:

```
---
- hosts: localhost
  tasks:
    - register: stack_out
      cloudformation:
        profile: slscode
        region: us-west-2
        stack_name: testStack
        template: ./cloudformation/simple-stack.json
        role_arn: arn:aws:iam::509803855674:role/bukkitcreate
```

Then, when run, here's the failure:

```
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'PhysicalResourceId'
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ryansb
/ansible_J1ty5m/ansible_module_cloudformation.py\", line 504, in <module>\n    main()\n  File \"/tmp/ryansb/ansible_J1ty5m/ansible_module_cl
oudformation.py\", line 467, in main\n    \"physical_resource_id\": res['PhysicalResourceId'],\nKeyError: 'PhysicalResourceId'\n", "module_s
tdout": "", "msg": "MODULE FAILURE"}
```

I'm working on a fix, I'll let you know when I get one. 

in_progress


Resolving patch:
diff --git a/cloud/amazon/cloudformation.py b/cloud/amazon/cloudformation.py
--- a/cloud/amazon/cloudformation.py
+++ b/cloud/amazon/cloudformation.py
@@ -464,7 +464,7 @@ def main():
         for res in reslist.get('StackResourceSummaries', []):
             stack_resources.append({
                 "logical_resource_id": res['LogicalResourceId'],
-                "physical_resource_id": res['PhysicalResourceId'],
+                "physical_resource_id": res.get('PhysicalResourceId', ''),
                 "resource_type": res['ResourceType'],
                 "last_updated_time": res['LastUpdatedTimestamp'],
                 "status": res['ResourceStatus'],


Test patch: