Repository: aws/aws-cli

Issue:
aws cloudformation deploy command could not wait stack create/update more than 10 minutes
Hi,

I am doing some tests by my side and I've run the aws cli in debug mode trying to find why the aws cli commands hangs up on long CloudFormation create/update processes,  I could find that the 'aws cloudformation deploy' command checks the CloudFormation Stack status every 5 seconds and then, after approximately 10 minutes and 30 seconds the code returns the error:

--- Error ---
"WaiterError: Waiter StackCreateComplete failed: Max attempts exceeded"
--- Error ---

I think the check period could be greater, then the aws cli command could wait more time, or increase the number of max attempts, but I think the first option it's better. Could you please take a look at this?

tests:

> time aws cloudformation deploy --template-file cfn-rolling-deployment.yaml --stack-name rollindeployment-test --capabilities CAPABILITY_NAMED_IAM
Waiting for changeset to be created..
Waiting for stack create/update to complete

Failed to create/update the stack. Run the following command
to fetch the list of events leading up to the failure
aws cloudformation describe-stack-events --stack-name rollindeployment-test
aws cloudformation deploy --template-file cfn-rolling-deployment.yaml          0.90s user 0.13s system 0% cpu 10:29.37 total

---
Traceback:
---
2017-08-09 10:48:41,571 - MainThread - botocore.hooks - DEBUG - Event needs-retry.cloudformation.DescribeStacks: calling handler <botocore.retryhandler.RetryHandler object at 0x10e3ac550>
2017-08-09 10:48:41,571 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2017-08-09 10:48:41,571 - MainThread - awscli.customizations.cloudformation.deployer - DEBUG - Execute changeset waiter exception
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/cloudformation/deployer.py", line 177, in wait_for_execute
    waiter.wait(StackName=stack_name)
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/botocore/waiter.py", line 327, in wait
    last_response=response
WaiterError: Waiter StackCreateComplete failed: Max attempts exceeded
2017-08-09 10:48:41,573 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/clidriver.py", line 200, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/clidriver.py", line 338, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/cloudformation/deploy.py", line 160, in _run_main
    parsed_args.execute_changeset)
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/cloudformation/deploy.py", line 172, in deploy
    deployer.wait_for_execute(stack_name, result.changeset_type)
  File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/cloudformation/deployer.py", line 181, in wait_for_execute
    raise exceptions.DeployFailedError(stack_name=stack_name)
DeployFailedError: Failed to create/update the stack. Run the following command
to fetch the list of events leading up to the failure
aws cloudformation describe-stack-events --stack-name rollindeployment-test2
2017-08-09 10:48:41,575 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Failed to create/update the stack. Run the following command
to fetch the list of events leading up to the failure
aws cloudformation describe-stack-events --stack-name rollindeployment-test2
aws cloudformation deploy --template-file cfn-rolling-deployment.yaml      0   1.21s user 0.16s system 0% cpu 10:31.42 total

Changed waiter.config.delay from 5 seconds to 10 seconds
The documentation below mentions the following with regards to: aws cloudformation wait stack-create-complete [1]
[1] http://docs.aws.amazon.com/cli/latest/reference/cloudformation/wait/stack-create-complete.html

"Wait until stack status is CREATE_COMPLETE. It will poll every 30 seconds until a successful state has been reached. 
This will exit with a return code of 255 after 120 failed checks."

This should be technically be 30 seconds not 5 seconds or 10 seconds because: 
120 calls at 30 seconds intervals will equal 3600 seconds/1 hour until the "Max attempts exceeded" which to confirms with CloudFormations 1 hour* timeout.

I understand that it is a fine line between performance (speed) and consistency, but unfortunately some services like CloudFront take time to build.


Discussion:
It happens on all versions past 1.11.117.

Any cloudformation deploy fails at 10 min mark.
Yes, this bug was introduced in this PR https://github.com/aws/aws-cli/pull/2686. Which lines up with the past 1.11.117 claim. More discussion and a potential fix is over there.
The issue is because the poller happens every 5 seconds waiting for a CREATE_COMPLETE 

In the below pull request you will find all the information 
https://github.com/aws/aws-cli/pull/2761


Resolving patch:
diff --git a/awscli/customizations/cloudformation/deployer.py b/awscli/customizations/cloudformation/deployer.py
--- a/awscli/customizations/cloudformation/deployer.py
+++ b/awscli/customizations/cloudformation/deployer.py
@@ -132,9 +132,10 @@ def wait_for_changeset(self, changeset_id, stack_name):
         # Wait for changeset to be created
         waiter = self._client.get_waiter("change_set_create_complete")
         # Poll every 5 seconds. Changeset creation should be fast
-        waiter.config.delay = 5
+        waiter_config = {'Delay': 5}
         try:
-            waiter.wait(ChangeSetName=changeset_id, StackName=stack_name)
+            waiter.wait(ChangeSetName=changeset_id, StackName=stack_name,
+                        WaiterConfig=waiter_config)
         except botocore.exceptions.WaiterError as ex:
             LOG.debug("Create changeset waiter exception", exc_info=ex)
 
@@ -178,10 +179,13 @@ def wait_for_execute(self, stack_name, changeset_type):
 
         # Poll every 5 seconds. Optimizing for the case when the stack has only
         # minimal changes, such the Code for Lambda Function
-        waiter.config.delay = 5
+        waiter_config = {
+            'Delay': 5,
+            'MaxAttempts': 720,
+        }
 
         try:
-            waiter.wait(StackName=stack_name)
+            waiter.wait(StackName=stack_name, WaiterConfig=waiter_config)
         except botocore.exceptions.WaiterError as ex:
             LOG.debug("Execute changeset waiter exception", exc_info=ex)
 


Test patch:
diff --git a/tests/unit/customizations/cloudformation/test_deployer.py b/tests/unit/customizations/cloudformation/test_deployer.py
--- a/tests/unit/customizations/cloudformation/test_deployer.py
+++ b/tests/unit/customizations/cloudformation/test_deployer.py
@@ -254,8 +254,10 @@ def test_wait_for_changeset_no_changes(self):
         with self.assertRaises(exceptions.ChangeEmptyError):
             mock_deployer.wait_for_changeset(changeset_id, stack_name)
 
+        waiter_config = {'Delay': 5}
         mock_waiter.wait.assert_called_once_with(ChangeSetName=changeset_id,
-                                                 StackName=stack_name)
+                                                 StackName=stack_name,
+                                                 WaiterConfig=waiter_config)
 
         mock_client.get_waiter.assert_called_once_with(
                 "change_set_create_complete")
@@ -282,8 +284,10 @@ def test_wait_for_changeset_failed_to_create_changeset(self):
         with self.assertRaises(RuntimeError):
             mock_deployer.wait_for_changeset(changeset_id, stack_name)
 
+        waiter_config = {'Delay': 5}
         mock_waiter.wait.assert_called_once_with(ChangeSetName=changeset_id,
-                                            StackName=stack_name)
+                                                 StackName=stack_name,
+                                                 WaiterConfig=waiter_config)
 
         mock_client.get_waiter.assert_called_once_with(
                 "change_set_create_complete")
@@ -305,7 +309,12 @@ def test_wait_for_execute_no_changes(self):
         with self.assertRaises(exceptions.DeployFailedError):
             mock_deployer.wait_for_execute(stack_name, changeset_type)
 
-        mock_waiter.wait.assert_called_once_with(StackName=stack_name)
+        waiter_config = {
+            'Delay': 5,
+            'MaxAttempts': 720,
+        }
+        mock_waiter.wait.assert_called_once_with(StackName=stack_name,
+                                                 WaiterConfig=waiter_config)
 
         mock_client.get_waiter.assert_called_once_with(
                 "stack_create_complete")