Repository: aws/aws-cli

Issue:
Allow aws cloudformation deploy to use template-url
As the [How to create serverless applications using AWS SAM](https://github.com/awslabs/serverless-application-model/blob/master/HOWTO.md) says : 

> SAM template is deployed to AWS CloudFormation by creating a changeset using the SAM template followed by executing the changeset... ...Alternatively, you can use aws cloudformation deploy CLI command to deploy the SAM template. Under-the-hood it creates and executes a changeset and waits until the deployment completes.

Nevertheless, a Cloudformation [create-change-set](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-change-set.html) allows to use the `--template-url` parameter, however the `aws cloudformation deploy` command does not. Please update the CLI to support `--template-url` while doing a `deploy`.

This is a feature already available on the AWS Management Console, where  you can deploy your stack using the URL that points to a template that is located in an S3 bucket.

AWS CLI version: 1.11.46


Discussion:
Marking as a feature request. @sanathkr Any thoughts on this?
Possible, but the problem is `deploy` command does a bit more than create-change-set. It allows you to specify only certain parameters that you'd like to override, instead of all the parameters in your template. It reads your template and passes UsePreviousValue=True for parameters that have not been overridden. If your template is in S3, we have to pull it down, which the CLI might not have permissions for.
The lack of `--template-url` support means that `aws cloudformation deploy` templates are limited to 51200 bytes in size. 

Even if `--template-url` wouldn't allow UsePreviousValue that would be a useful feature.
Is there any timeline for this feature or a workaround available(other than splitting the template)? My template is affected by the limit.
One workaround is to split the template.

Alternatively, you can rewrite the deploy logic using AWS CLI.  You just need to:

- Check for the existence of the stack `aws cloudformation describe-stacks` - and parse the stderr for "does not exist"
- Create a change-set `aws cloudformation create-change-set` - using --change-set-type CREATE or UPDATE appropriately
- Wait for creation to complete `aws cloudformation wait change-set-create-complete`
- Do an `aws cloudformation describe-change-set` and check the StatusReason for "The submitted information didn't contain changes"
- Conditionally execute the change-set `aws cloudformation execute-change-set`. Don't execute no-op changesets.
- Wait for the change-set to execute. `aws cloudformation wait change-set-create-complete` waits for the CREATE_COMPLETE  status and behaves badly in case of failure (takes 1 hour to time-out). Instead, write a loop to poll `aws cloudformation describe-stacks` for the stack status. There are [16 possible statuses](https://github.com/rvedotrc/aws-cloudformation-stack-states)


The lack of this also makes it hard to follow https://aws.amazon.com/blogs/devops/aws-cloudformation-security-best-practices/ and restrict access to a specific template. 
Yeah, this is definitely an issue we should fix. this requires quite a bit of changes to the code. PRs appreciated!

Instead of reading a local file, AWSCLI will pull the template from given S3 location, parse the parameters out, merge with the parameter overrides arguments, and call create-change-set with S3 template URL instead of uploading the template text

Resolving patch:
diff --git a/awscli/customizations/cloudformation/deploy.py b/awscli/customizations/cloudformation/deploy.py
--- a/awscli/customizations/cloudformation/deploy.py
+++ b/awscli/customizations/cloudformation/deploy.py
@@ -15,8 +15,11 @@
 import sys
 import logging
 
+from botocore.client import Config
+
 from awscli.customizations.cloudformation import exceptions
 from awscli.customizations.cloudformation.deployer import Deployer
+from awscli.customizations.cloudformation.s3uploader import S3Uploader
 from awscli.customizations.cloudformation.yamlhelper import yaml_parse
 
 from awscli.customizations.commands import BasicCommand
@@ -61,6 +64,41 @@ class DeployCommand(BasicCommand):
                 ' stack. If you specify a new stack, the command creates it.'
             )
         },
+        {
+            'name': 's3-bucket',
+            'required': False,
+            'help_text': (
+                'The name of the S3 bucket where this command uploads your '
+                'CloudFormation template. This is required the deployments of '
+                'templates sized greater than 51,200 bytes'
+            )
+        },
+        {
+            "name": "force-upload",
+            "action": "store_true",
+            "help_text": (
+                'Indicates whether to override existing files in the S3 bucket.'
+                ' Specify this flag to upload artifacts even if they '
+                ' match existing artifacts in the S3 bucket.'
+            )
+        },
+        {
+            'name': 's3-prefix',
+            'help_text': (
+                'A prefix name that the command adds to the'
+                ' artifacts\' name when it uploads them to the S3 bucket.'
+                ' The prefix name is a path name (folder name) for'
+                ' the S3 bucket.'
+            )
+        },
+
+        {
+            'name': 'kms-key-id',
+            'help_text': (
+                'The ID of an AWS KMS key that the command uses'
+                ' to encrypt artifacts that are at rest in the S3 bucket.'
+            )
+        },
         {
             'name': PARAMETER_OVERRIDE_CMD,
             'action': 'store',
@@ -177,22 +215,45 @@ def _run_main(self, parsed_args, parsed_globals):
 
         parameters = self.merge_parameters(template_dict, parameter_overrides)
 
+        template_size = os.path.getsize(parsed_args.template_file)
+        if template_size > 51200 and not parsed_args.s3_bucket:
+            raise exceptions.DeployBucketRequiredError()
+
+        bucket = parsed_args.s3_bucket
+        if bucket:
+            s3_client = self._session.create_client(
+                "s3",
+                config=Config(signature_version='s3v4'),
+                region_name=parsed_globals.region,
+                verify=parsed_globals.verify_ssl)
+
+            s3_uploader = S3Uploader(s3_client,
+                                      bucket,
+                                      parsed_globals.region,
+                                      parsed_args.s3_prefix,
+                                      parsed_args.kms_key_id,
+                                      parsed_args.force_upload)
+        else:
+            s3_uploader = None
+
         deployer = Deployer(cloudformation_client)
         return self.deploy(deployer, stack_name, template_str,
                            parameters, parsed_args.capabilities,
                            parsed_args.execute_changeset, parsed_args.role_arn,
-                           parsed_args.notification_arns)
+                           parsed_args.notification_arns, s3_uploader)
+
 
     def deploy(self, deployer, stack_name, template_str,
                parameters, capabilities, execute_changeset, role_arn,
-               notification_arns):
+               notification_arns, s3_uploader):
         result = deployer.create_and_wait_for_changeset(
                 stack_name=stack_name,
                 cfn_template=template_str,
                 parameter_values=parameters,
                 capabilities=capabilities,
                 role_arn=role_arn,
-                notification_arns=notification_arns)
+                notification_arns=notification_arns,
+                s3_uploader=s3_uploader)
 
         if execute_changeset:
             deployer.execute_changeset(result.changeset_id, stack_name)
diff --git a/awscli/customizations/cloudformation/deployer.py b/awscli/customizations/cloudformation/deployer.py
--- a/awscli/customizations/cloudformation/deployer.py
+++ b/awscli/customizations/cloudformation/deployer.py
@@ -18,6 +18,8 @@
 import collections
 
 from awscli.customizations.cloudformation import exceptions
+from awscli.customizations.cloudformation.artifact_exporter import mktempfile, parse_s3_url
+
 from datetime import datetime
 
 LOG = logging.getLogger(__name__)
@@ -71,7 +73,7 @@ def has_stack(self, stack_name):
 
     def create_changeset(self, stack_name, cfn_template,
                          parameter_values, capabilities, role_arn,
-                         notification_arns):
+                         notification_arns, s3_uploader):
         """
         Call Cloudformation to create a changeset and wait for it to complete
 
@@ -106,6 +108,19 @@ def create_changeset(self, stack_name, cfn_template,
             'Capabilities': capabilities,
             'Description': description,
         }
+
+        # If an S3 uploader is available, use TemplateURL to deploy rather than
+        # TemplateBody. This is required for large templates.
+        if s3_uploader:
+            with mktempfile() as temporary_file:
+                temporary_file.write(kwargs.pop('TemplateBody'))
+                temporary_file.flush()
+                url = s3_uploader.upload_with_dedup(
+                        temporary_file.name, "template")
+                # TemplateUrl property requires S3 URL to be in path-style format
+                parts = parse_s3_url(url, version_property="Version")
+                kwargs['TemplateURL'] = s3_uploader.to_path_style_s3_url(parts["Key"], parts.get("Version", None))
+
         # don't set these arguments if not specified to use existing values
         if role_arn is not None:
             kwargs['RoleARN'] = role_arn
@@ -126,7 +141,7 @@ def wait_for_changeset(self, changeset_id, stack_name):
         :param stack_name:   Stack name
         :return: Latest status of the create-change-set operation
         """
-        sys.stdout.write("Waiting for changeset to be created..\n")
+        sys.stdout.write("\nWaiting for changeset to be created..\n")
         sys.stdout.flush()
 
         # Wait for changeset to be created
@@ -189,11 +204,11 @@ def wait_for_execute(self, stack_name, changeset_type):
 
     def create_and_wait_for_changeset(self, stack_name, cfn_template,
                                       parameter_values, capabilities, role_arn,
-                                      notification_arns):
+                                      notification_arns, s3_uploader):
 
         result = self.create_changeset(
                 stack_name, cfn_template, parameter_values, capabilities,
-                role_arn, notification_arns)
+                role_arn, notification_arns, s3_uploader)
 
         self.wait_for_changeset(result.changeset_id, stack_name)
 
diff --git a/awscli/customizations/cloudformation/exceptions.py b/awscli/customizations/cloudformation/exceptions.py
--- a/awscli/customizations/cloudformation/exceptions.py
+++ b/awscli/customizations/cloudformation/exceptions.py
@@ -53,3 +53,10 @@ class DeployFailedError(CloudFormationCommandError):
          "to fetch the list of events leading up to the failure"
          "\n"
          "aws cloudformation describe-stack-events --stack-name {stack_name}")
+
+class DeployBucketRequiredError(CloudFormationCommandError):
+    fmt = \
+        ("Templates with a size greater than 51,200 bytes must be deployed "
+         "via an S3 Bucket. Please add the --s3-bucket parameter to your "
+         "command. The local template will be copied to that S3 bucket and "
+         "then deployed.")


Test patch:
diff --git a/tests/unit/customizations/cloudformation/test_deploy.py b/tests/unit/customizations/cloudformation/test_deploy.py
--- a/tests/unit/customizations/cloudformation/test_deploy.py
+++ b/tests/unit/customizations/cloudformation/test_deploy.py
@@ -56,7 +56,8 @@ def setUp(self):
                                     execute_changeset=True,
                                     capabilities=None,
                                     role_arn=None,
-                                    notification_arns=[])
+                                    notification_arns=[],
+                                    s3_bucket=None)
         self.parsed_globals = FakeArgs(region="us-east-1", endpoint_url=None,
                                        verify_ssl=None)
         self.deploy_command = DeployCommand(self.session)
@@ -109,7 +110,8 @@ def test_command_invoked(self, mock_yaml_parse):
                         None,
                         not self.parsed_args.no_execute_changeset,
                         None,
-                        [])
+                        [],
+                        mock.ANY)
 
                 self.deploy_command.parse_parameter_arg.assert_called_once_with(
                         self.parsed_args.parameter_overrides)
@@ -125,6 +127,27 @@ def test_invalid_template_file(self):
             result = self.deploy_command._run_main(self.parsed_args,
                                                   parsed_globals=self.parsed_globals)
 
+    @patch('awscli.customizations.cloudformation.deploy.os.path.isfile')
+    @patch('awscli.customizations.cloudformation.deploy.yaml_parse')
+    @patch('awscli.customizations.cloudformation.deploy.os.path.getsize')
+    def test_s3_upload_required_but_missing_bucket(self, mock_getsize, mock_yaml_parse, mock_isfile):
+        """
+        Tests that large templates are detected prior to deployment
+        """
+        template_str = get_example_template()
+
+        mock_getsize.return_value = 51201
+        mock_isfile.return_value = True
+        mock_yaml_parse.return_value = template_str
+        open_mock = mock.mock_open()
+
+        with patch(
+                "awscli.customizations.cloudformation.deploy.open",
+                open_mock(read_data=template_str)) as open_mock:
+            with self.assertRaises(exceptions.DeployBucketRequiredError):
+                result = self.deploy_command._run_main(self.parsed_args,
+                                parsed_globals=self.parsed_globals)
+
 
     def test_deploy_success(self):
         """
@@ -140,7 +163,7 @@ def test_deploy_success(self):
         changeset_type = "CREATE"
         role_arn = "arn:aws:iam::1234567890:role"
         notification_arns = ["arn:aws:sns:region:1234567890:notify"]
-
+        s3_uploader = None
 
         # Set the mock to return this fake changeset_id
         self.deployer.create_and_wait_for_changeset.return_value = ChangeSetResult(changeset_id, changeset_type)
@@ -152,7 +175,8 @@ def test_deploy_success(self):
                                    capabilities,
                                    execute_changeset,
                                    role_arn,
-                                   notification_arns)
+                                   notification_arns,
+                                   s3_uploader)
         self.assertEqual(rc, 0)
 
 
@@ -161,7 +185,8 @@ def test_deploy_success(self):
                                                      parameter_values=parameters,
                                                      capabilities=capabilities,
                                                      role_arn=role_arn,
-                                                     notification_arns=notification_arns)
+                                                     notification_arns=notification_arns,
+                                                     s3_uploader=s3_uploader)
 
         # since execute_changeset is set to True, deploy() will execute changeset
         self.deployer.execute_changeset.assert_called_once_with(changeset_id, stack_name)
@@ -177,6 +202,7 @@ def test_deploy_no_execute(self):
         execute_changeset = False
         role_arn = "arn:aws:iam::1234567890:role"
         notification_arns = ["arn:aws:sns:region:1234567890:notify"]
+        s3_uploader = None
 
 
         self.deployer.create_and_wait_for_changeset.return_value = ChangeSetResult(changeset_id, "CREATE")
@@ -187,7 +213,8 @@ def test_deploy_no_execute(self):
                                             capabilities,
                                             execute_changeset,
                                             role_arn,
-                                            notification_arns)
+                                            notification_arns,
+                                            s3_uploader)
         self.assertEqual(rc, 0)
 
         self.deployer.create_and_wait_for_changeset.assert_called_once_with(stack_name=stack_name,
@@ -195,7 +222,8 @@ def test_deploy_no_execute(self):
                                                      parameter_values=parameters,
                                                      capabilities=capabilities,
                                                      role_arn=role_arn,
-                                                     notification_arns=notification_arns)
+                                                     notification_arns=notification_arns,
+                                                     s3_uploader=s3_uploader)
 
         # since execute_changeset is set to True, deploy() will execute changeset
         self.deployer.execute_changeset.assert_not_called()
@@ -210,7 +238,7 @@ def test_deploy_raise_exception(self):
         execute_changeset = True
         role_arn = "arn:aws:iam::1234567890:role"
         notification_arns = ["arn:aws:sns:region:1234567890:notify"]
-
+        s3_uploader = None
 
         self.deployer.wait_for_execute.side_effect = RuntimeError("Some error")
         with self.assertRaises(RuntimeError):
@@ -221,7 +249,8 @@ def test_deploy_raise_exception(self):
                                        capabilities,
                                        execute_changeset,
                                        role_arn,
-                                       notification_arns)
+                                       notification_arns,
+                                       s3_uploader)
 
 
     def test_parse_parameter_arg_success(self):
diff --git a/tests/unit/customizations/cloudformation/test_deployer.py b/tests/unit/customizations/cloudformation/test_deployer.py
--- a/tests/unit/customizations/cloudformation/test_deployer.py
+++ b/tests/unit/customizations/cloudformation/test_deployer.py
@@ -103,6 +103,7 @@ def test_create_changeset_success(self):
         capabilities = ["capabilities"]
         role_arn = "arn:aws:iam::1234567890:role"
         notification_arns = ["arn:aws:sns:region:1234567890:notify"]
+        s3_uploader = None
 
         # Case 1: Stack DOES NOT exist
         self.deployer.has_stack = Mock()
@@ -129,7 +130,7 @@ def test_create_changeset_success(self):
         with self.stub_client:
             result = self.deployer.create_changeset(
                     stack_name, template, parameters, capabilities, role_arn,
-                    notification_arns)
+                    notification_arns, s3_uploader)
             self.assertEquals(response["Id"], result.changeset_id)
             self.assertEquals("CREATE", result.changeset_type)
 
@@ -142,7 +143,76 @@ def test_create_changeset_success(self):
         with self.stub_client:
             result = self.deployer.create_changeset(
                     stack_name, template, parameters, capabilities, role_arn,
-                    notification_arns)
+                    notification_arns, s3_uploader)
+            self.assertEquals(response["Id"], result.changeset_id)
+            self.assertEquals("UPDATE", result.changeset_type)
+
+    def test_create_changeset_success_s3_bucket(self):
+        stack_name = "stack_name"
+        template = "template"
+        template_url = "https://s3.amazonaws.com/bucket/file"
+        parameters = [
+            {"ParameterKey": "Key1", "ParameterValue": "Value"},
+            {"ParameterKey": "Key2", "UsePreviousValue": True},
+            {"ParameterKey": "Key3", "UsePreviousValue": False},
+        ]
+        # Parameters that Use Previous Value will be removed on stack creation
+        # to either force CloudFormation to use the Default value, or ask user to specify a parameter
+        filtered_parameters = [
+            {"ParameterKey": "Key1", "ParameterValue": "Value"},
+            {"ParameterKey": "Key3", "UsePreviousValue": False},
+        ]
+        capabilities = ["capabilities"]
+        role_arn = "arn:aws:iam::1234567890:role"
+        notification_arns = ["arn:aws:sns:region:1234567890:notify"]
+
+        s3_uploader = Mock()
+        def to_path_style_s3_url(some_string, Version=None):
+            return "https://s3.amazonaws.com/bucket/file"
+        s3_uploader.to_path_style_s3_url = to_path_style_s3_url
+        def upload_with_dedup(filename,extension):
+            return "s3://bucket/file"
+        s3_uploader.upload_with_dedup = upload_with_dedup
+
+        # Case 1: Stack DOES NOT exist
+        self.deployer.has_stack = Mock()
+        self.deployer.has_stack.return_value = False
+
+        expected_params = {
+            "ChangeSetName": botocore.stub.ANY,
+            "StackName": stack_name,
+            "TemplateURL": template_url,
+            "ChangeSetType": "CREATE",
+            "Parameters": filtered_parameters,
+            "Capabilities": capabilities,
+            "Description": botocore.stub.ANY,
+            "RoleARN": role_arn,
+            "NotificationARNs": notification_arns
+        }
+
+        response = {
+            "Id": "changeset ID"
+        }
+
+        self.stub_client.add_response("create_change_set", response,
+                                      expected_params)
+        with self.stub_client:
+            result = self.deployer.create_changeset(
+                    stack_name, template, parameters, capabilities, role_arn,
+                    notification_arns, s3_uploader)
+            self.assertEquals(response["Id"], result.changeset_id)
+            self.assertEquals("CREATE", result.changeset_type)
+
+        # Case 2: Stack exists. We are updating it
+        self.deployer.has_stack.return_value = True
+        expected_params["ChangeSetType"] = "UPDATE"
+        expected_params["Parameters"] = parameters
+        self.stub_client.add_response("create_change_set", response,
+                                      expected_params)
+        with self.stub_client:
+            result = self.deployer.create_changeset(
+                    stack_name, template, parameters, capabilities, role_arn,
+                    notification_arns, s3_uploader)
             self.assertEquals(response["Id"], result.changeset_id)
             self.assertEquals("UPDATE", result.changeset_type)
 
@@ -154,6 +224,7 @@ def test_create_changeset_exception(self):
         capabilities = ["capabilities"]
         role_arn = "arn:aws:iam::1234567890:role"
         notification_arns = ["arn:aws:sns:region:1234567890:notify"]
+        s3_uploader = None
 
         self.deployer.has_stack = Mock()
         self.deployer.has_stack.return_value = False
@@ -163,7 +234,7 @@ def test_create_changeset_exception(self):
         with self.stub_client:
             with self.assertRaises(botocore.exceptions.ClientError):
                 self.deployer.create_changeset(stack_name, template, parameters,
-                capabilities, role_arn, notification_arns)
+                capabilities, role_arn, notification_arns, None)
 
     def test_execute_changeset(self):
         stack_name = "stack_name"
@@ -198,6 +269,7 @@ def test_create_and_wait_for_changeset_successful(self):
         changeset_type = "changeset type"
         role_arn = "arn:aws:iam::1234567890:role"
         notification_arns = ["arn:aws:sns:region:1234567890:notify"]
+        s3_uploader = None
 
         self.deployer.create_changeset = Mock()
         self.deployer.create_changeset.return_value = ChangeSetResult(changeset_id, changeset_type)
@@ -206,7 +278,7 @@ def test_create_and_wait_for_changeset_successful(self):
 
         result = self.deployer.create_and_wait_for_changeset(
                 stack_name, template, parameters, capabilities, role_arn,
-                notification_arns)
+                notification_arns, s3_uploader)
         self.assertEquals(result.changeset_id, changeset_id)
         self.assertEquals(result.changeset_type, changeset_type)
 
@@ -220,6 +292,7 @@ def test_create_and_wait_for_changeset_error_waiting_for_changeset(self):
         changeset_type = "changeset type"
         role_arn = "arn:aws:iam::1234567890:role"
         notification_arns = ["arn:aws:sns:region:1234567890:notify"]
+        s3_uploader = None
 
         self.deployer.create_changeset = Mock()
         self.deployer.create_changeset.return_value = ChangeSetResult(changeset_id, changeset_type)
@@ -230,7 +303,7 @@ def test_create_and_wait_for_changeset_error_waiting_for_changeset(self):
         with self.assertRaises(RuntimeError):
             result = self.deployer.create_and_wait_for_changeset(
                     stack_name, template, parameters, capabilities, role_arn,
-                    notification_arns)
+                    notification_arns, s3_uploader)
 
     def test_wait_for_changeset_no_changes(self):
         stack_name = "stack_name"