Repository: ansible/ansible

Issue:
cloudformation create_change_set fails when stack is in "  
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
cloudformation.py

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0 (devel 0343ac1a9d) last updated 2017/08/06 13:45:21 (GMT +000)
  config file = /saas-aws-deployment/cf-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /ansible/lib/ansible
  executable location = /ansible/bin/ansible
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
N/A
##### SUMMARY
<!--- Explain the problem briefly -->
When stack name is in "UPDATE_ROLLBACK_COMPLETE" state,
the changeset created successfully but Ansible exists with fatal error

When executing exactly the same CF template on the same stack, the command succeed.
The same if calling boto3 directly.

When executing exactly the same CF template on the same stack which is in "UPDATE_COMPLETE" state, the command succeed.  

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Put stack into "UPDATE_ROLLBACK_COMPLETE" state and execute create changeset.
<!--- Paste example playbooks or commands between quotes below -->
```
- name: Creating CF changeset
  cloudformation:
    stack_name:   "{{stack_name}}"
    profile:      "{{profile}}"
    region:       "{{region}}"
    state:        "{{stack_state}}"
    template_url: "{{s3_directory}}/{{profile}}/{{template}}"
    changeset_name: "{{stack_name}}-{{revision_cf}}"
    create_changeset: true
    tags:
      Revision:   "{{revision}}"
  register:     "{{stack_name}}"
  when: stack_state == "present"
  tags:
    - deploy
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
[WARNING]: Created changeset named XX for stack XX
[WARNING]: You can execute it using: aws cloudformation execute-change-set --change-set-name XX
[WARNING]: NOTE that dependencies on this stack might fail due to pending changes!

changed: [localhost] => {
    "changed": true,
    "events": [
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE_CLEANUP_IN_PROGRESS",
        "StackEvent AWS::RDS::DBInstance XX UPDATE_COMPLETE",
        "StackEvent AWS::RDS::DBInstance XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE_CLEANUP_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX CREATE_COMPLETE",
        "StackEvent AWS::RDS::DBInstance XX CREATE_COMPLETE",
        "StackEvent AWS::RDS::DBInstance XX CREATE_IN_PROGRESS",
        "StackEvent AWS::RDS::DBInstance XX CREATE_IN_PROGRESS",
        "StackEvent AWS::RDS::DBSubnetGroup XX CREATE_COMPLETE",
        "StackEvent AWS::RDS::DBSubnetGroup XX CREATE_IN_PROGRESS",
        "StackEvent AWS::RDS::DBSubnetGroup XX CREATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX CREATE_IN_PROGRESS"
    ],
    "failed": false,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_secret_key": null,
            "changeset_name": "XX-10",
            "create_changeset": true,
            "disable_rollback": false,
            "ec2_url": null,
            "notification_arns": null,
            "profile": "test",
            "region": "us-west-2",
            "role_arn": null,
            "security_token": null,
            "stack_name": "XX",
            "stack_policy": null,
            "state": "present",
            "tags": {
                "Revision": "1.0"
            },
            "template": null,
            "template_format": null,
            "template_parameters": {},
            "template_url": "XX",
            "validate_certs": true
        }
    },
    "log": [],
    "output": "Stack UPDATE complete",
    "stack_outputs": {
 
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
 [WARNING]: Created changeset named XX for stack XX
 [WARNING]: You can execute it using: aws cloudformation execute-change-set --change-set-name XX
 [WARNING]: NOTE that dependencies on this stack might fail due to pending changes!

fatal: [localhost]: FAILED! => {
    "changed": true,
    "events": [
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_ROLLBACK_COMPLETE",
        "StackEvent AWS::IAM::Role TestSetChange DELETE_COMPLETE",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS",
        "StackEvent AWS::RDS::DBInstance XX UPDATE_COMPLETE",
        "StackEvent AWS::RDS::DBInstance XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_ROLLBACK_IN_PROGRESS",
        "StackEvent AWS::IAM::Role TestSetChange CREATE_FAILED",
        "StackEvent AWS::IAM::Role TestSetChange CREATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_ROLLBACK_COMPLETE",
        "StackEvent AWS::IAM::Role TestSetChange DELETE_COMPLETE",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS",
        "StackEvent AWS::RDS::DBInstance XX UPDATE_COMPLETE",
        "StackEvent AWS::RDS::DBInstance XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_ROLLBACK_IN_PROGRESS",
        "StackEvent AWS::IAM::Role TestSetChange CREATE_FAILED",
        "StackEvent AWS::IAM::Role TestSetChange CREATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE_CLEANUP_IN_PROGRESS",
        "StackEvent AWS::RDS::DBInstance XX UPDATE_COMPLETE",
        "StackEvent AWS::RDS::DBInstance XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE_CLEANUP_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_COMPLETE_CLEANUP_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX UPDATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX CREATE_COMPLETE",
        "StackEvent AWS::RDS::DBInstance XX CREATE_COMPLETE",
        "StackEvent AWS::RDS::DBInstance XX CREATE_IN_PROGRESS",
        "StackEvent AWS::RDS::DBInstance XX CREATE_IN_PROGRESS",
        "StackEvent AWS::RDS::DBSubnetGroup XX CREATE_COMPLETE",
        "StackEvent AWS::RDS::DBSubnetGroup XX CREATE_IN_PROGRESS",
        "StackEvent AWS::RDS::DBSubnetGroup XX CREATE_IN_PROGRESS",
        "StackEvent AWS::CloudFormation::Stack XX CREATE_IN_PROGRESS"
    ],
    "failed": true,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_secret_key": null,
            "changeset_name": "XX",
            "create_changeset": true,
            "disable_rollback": false,
            "ec2_url": null,
            "notification_arns": null,
            "profile": "dev",
            "region": "us-west-2",
            "role_arn": null,
            "security_token": null,
            "stack_name": "XX",
            "stack_policy": null,
            "state": "present",
            "tags": {
                "Revision": "1.0"
            },
            "template": null,
            "template_format": null,
            "template_parameters": {},
            "template_url": "XX",
            "validate_certs": true
        }
    },
    "output": "Problem with UPDATE. Rollback complete",

<!--- Paste verbatim command output between quotes below -->
```

```



Discussion:
cc @jsmartin @ryansb @tedder
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Hmm, @gregory-dan can you walk through the code? I bet you can figure out the code path that is failing. Lemme know if you need more help.

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/cloudformation.py b/lib/ansible/modules/cloud/amazon/cloudformation.py
--- a/lib/ansible/modules/cloud/amazon/cloudformation.py
+++ b/lib/ansible/modules/cloud/amazon/cloudformation.py
@@ -314,7 +314,7 @@ def create_changeset(module, stack_params, cfn):
             result = dict(changed=False, output='ChangeSet %s already exists.' % changeset_name, warnings=[warning])
         else:
             cs = cfn.create_change_set(**stack_params)
-            result = stack_operation(cfn, stack_params['StackName'], 'UPDATE')
+            result = stack_operation(cfn, stack_params['StackName'], 'CREATE_CHANGESET')
             result['warnings'] = ['Created changeset named %s for stack %s' % (changeset_name, stack_params['StackName']),
                 'You can execute it using: aws cloudformation execute-change-set --change-set-name %s' % cs['Id'],
                 'NOTE that dependencies on this stack might fail due to pending changes!']
@@ -378,7 +378,7 @@ def stack_operation(cfn, stack_name, operation):
                 return ret
         # it covers ROLLBACK_COMPLETE and UPDATE_ROLLBACK_COMPLETE
         # Possible states: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-describing-stacks.html#w1ab2c15c17c21c13
-        elif stack['StackStatus'].endswith('ROLLBACK_COMPLETE'):
+        elif stack['StackStatus'].endswith('ROLLBACK_COMPLETE') and operation != 'CREATE_CHANGESET':
             ret.update({'changed': True, 'failed': True, 'output': 'Problem with %s. Rollback complete' % operation})
             return ret
         # note the ordering of ROLLBACK_COMPLETE and COMPLETE, because otherwise COMPLETE will match both cases.


Test patch: