Repository: aws/aws-cli

Issue:
`aws cloudformation package` breaks certain string literals in templates
### CloudFormation template
#### `template.yml`
```yaml
Parameters:
  Value7:
    Type: String
    Default: "077777777777"
  Value8:
    Type: String
    Default: "088888888888"
  Value9:
    Type: String
    Default: "099999999999"
Resources:
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: ./template_child.yml
```

#### `template_child.yml`
```yaml
# This template is not so important
Resources:
  Dummy:
    Type: Custom::Whatever
    Properties:
      ServiceToken: whatever
      Value1: whatever
```

### Commands to reproduce
```console
$ uname -a
Darwin xxxxxxx 18.2.0 Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64 x86_64

$ aws --version
aws-cli/1.16.110 Python/3.7.2 Darwin/18.2.0 botocore/1.12.100

$ aws cloudformation package --s3-bucket BUCKET_NAME --template-file template.yml                                                           
Uploading to 137ce8b72427772da39a43ddc087908a.template  101 / 101.0  (100.00%)
Parameters:
  Value7:
    Type: String
    Default: '077777777777'
  Value8:
    Type: String
    Default: 088888888888
  Value9:
    Type: String
    Default: 099999999999
Resources:
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/BUCKET_NAME/137ce8b72427772da39a43ddc087908a.template
```

Quotes of `Value8` and `Value9` are removed after `aws cloudformation package`.

It seems that leads to the values interpreted as numbers during stack operation.
If I use the values for `!Sub`, values will be like `9.9999999999E10`, which is unintended and cause errors.


Discussion:
https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudformation/yamlhelper.py


```console 
$ python3
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> from awscli.customizations.cloudformation.yamlhelper import yaml_parse, yaml_dump

>>> parsed = yaml_parse('sevens: "077777777777"\neights: "088888888888"')

>>> parsed
OrderedDict([('sevens', '077777777777'), ('eights', '088888888888')])

>>> yaml_dump(parsed)
"sevens: '077777777777'\neights: 088888888888\n"
```

It seems quotes are erased during dumping to yaml.

```console
$ python3
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml

>>> yaml
<module 'yaml' from '/usr/local/Cellar/awscli/1.16.110/libexec/lib/python3.7/site-packages/yaml/__init__.py'>

>>> yaml.dump({"sevens": "077777777777", "eights": "088888888888"}, default_flow_style=False)

"eight: 088888888888\nseven: '077777777777'\n"
```

Seems a bug of PyYAML?:thinking:
I thought PyYAML generates invalid yaml, but PyYAML parses correctly (as string type)  without quotes.

```
>>> yaml.load("eights: 088888888888\nsevens: '077777777777'\n")
{'eights': '088888888888', 'sevens': '077777777777'}
```

I suspect either PyYAML or the yaml parser used in CloudFormation API is not following the yaml spec (related to octal base int type and string type without quotes) and causing this issue.

https://yaml.org/type/int.html

In any case, `aws cloudformation package` should generates yaml that can be used for CloudFormation.
Any updates?
Please let me know if you need further information.
I guessed that CloudFormation API uses SnakeYAML from its behavior, so I filed an issue for SnakeYAML.

https://bitbucket.org/snakeyaml/snakeyaml/issues/442

The SnakeYAML author says that it is a bug in YAML 1.1 spec...

I think aws-cli should dump strings in YAML with quotes for parser interoperability.
https://stackoverflow.com/questions/38369833/pyyaml-and-using-quotes-for-strings-only
Related?
https://github.com/yaml/pyyaml/issues/98
https://github.com/awslabs/aws-cfn-template-flip/issues/41
https://github.com/awslabs/aws-sam-cli/issues/867
We are looking into this issue.  @sanathkr, what are your thoughts?

We are running into a similar issue. We have Account ID as the key in template.yaml:

# template.yaml
```
...
Mappings:
  AccountMap:
    "098111111198":
      Env: "dev"
    "988111111198":
      Env: "test"
    "123456789012":
      Env: "prod"
...
```
After the `sam package --template-file template.yaml --output-template-file packaged.yaml     --s3-bucket BUCKET_NAME`, quotes around the first Account ID got stripped:

# packaged.yaml
```
...
Mappings:
  AccountMap:
    098111111198:
      Env: dev
    '988111111198':
      Env: test
    '123456789012':
      Env: prod
...
```
The subsequent `aws cloudformation deploy --template-file packaged.yaml --stack-name STACK_NAME ...`  thinks `098111111198` is a number and throws `An error occurred (ValidationError) when calling the CreateChangeSet operation: Template format error: [/Mappings/AccountMap] map keys must be strings; received numeric [9.8111111198E10] instead`.

```
$ sam --version
SAM CLI, version 0.17.0
$ aws --version
aws-cli/1.16.150 Python/3.7.3 Darwin/18.5.0 botocore/1.12.140
$ python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.9.172'
```
We would appreciate suggestions to any workarounds. Thanks!
Having the exact same issue outlined by @beibeiyang  above. Any updates?
I'm experiencing the same weird issue.  Very weird.  Things up to 7 are working, and 8 will break as per above issue.
I've also encountered the same issue (awscli 1.16.199), any updates or plans to produce updates?
Same issue here, when following instructions to create a bucket policy for us-east-2, using was-cli 1.16.260:

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#enable-access-logging
I've switched to use [AWS CDK](https://github.com/aws/aws-cdk) after all.
CDK is not affected by this issue, I suppose.
@justnance Is this still being worked on?  Is there a workaround?
This breaks the creation of Cognito UserPool because the MFAConfiguration requires quotes surrounding the "OFF" value these are stripped off by the cli . Is there any workaround to force quotes ?
@idm-ryou Did you find a workaround in the sam template for account_id map lookup?
@lmayorga1980  I found a workaround.  I had to move the reference to the offending numerical string to the top level template file (since I was using it in a nested stack), and then add `--use-json` to the `aws cloudformation package command.  This will cause the top level stack to be output as json instead of yaml.
Today I faced strange error with templates and parameters which were working without any issues:
```
An error occurred (ValidationError) when calling the CreateStack operation: Parameter 'EnableHA' must be one of AllowedValues
```

This `EnableHA` has only two allowed values: `"yes"` and `"no"`, and my parameters json file defines it to `"no"`.

Investigation showed that aws-cli 2 (2.0.7 and 2.0.9) converts these strings to just `yes` and `no` which have special meaning in YAML and so my supplied parameters don't pass validation.

This should be fixed ASAP.
We were having the same problem in our SAM template since our account starts with a '0' as well. I was able to get around it by using `!Ref 'AWS::AccountId'` where I needed the account ID.

See here: [https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html)
I have just run into the same issue when trying to stand up one AWS genomics' cloudformation stacks. It uses a parameter of type `String` whose `AllowedValues` are `Yes` and `No`. When I attempt to pass one of those strings using awscli2, I get a message that I must use one of the AllowedValues. It doesn't matter whether the value is quote when I pass the `ParameterValue`.  Interestingly, when I run `validate-template`, this is in the output:
```
{
            "ParameterKey": "ExistingBucket",
            "DefaultValue": "false",
            "NoEcho": false,
            "Description": "Does this bucket already exist?"
}
```

So it appears that at some point yes and no become true and false. The workaround is to just pass the strings `true` or `false`.

Experiencing the same issue with account IDs as Map keys.
This causes problems for a string with colons like so:
```YAML
Resources:
  WindowsFileSystemWithAllConfigs:
    Type: "AWS::FSx::FileSystem"
    # ...
      WindowsConfiguration:
        WeeklyMaintenanceStartTime: "4:16:30"
        DailyAutomaticBackupStartTime: "01:00"
        # ...
```

It will be come `WeeklyMaintenanceStartTime: 4:16:30`, which is then interpreted as a number and violates the field validation rule. 

Example error while deploying:

> 2 validation errors detected: Value '15390' at 'windowsConfiguration.weeklyMaintenanceStartTime' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[1-7]:([01]\d|2[0-3]):?([0-5]\d)$; Value '15390' at 'windowsConfiguration.weeklyMaintenanceStartTime' failed to satisfy constraint: Member must have length greater than or equal to 7 (Service: AmazonFSx; Status Code: 400; Error Code: BadRequest; Request ID: cbeff8a0-ce03-452c-9f0a-f8e08d894dc9; Proxy: null)

We’re running into this on our end as well. It's surprising and concerning that this issue is 2 years old is still unaddressed.
I just ran into this issue to seems sporadic as well was working and all of a sudden stopped working on a template that was working
I have encountered the same issue.

I can replicate this issue on v 1.19.97 of the CLI
I cannot replicate it on v2.2.12
Encounter this issue in aws-cli/2.7.3 Python/3.9.11. The single quote from Time being remove by package for AWS::DLM::LifecyclePlicy PolicyDetails Schedules CreateRules. 

Submitted Template:
`
            CreateRule:
              Interval: 12
              IntervalUnit: HOURS
              Times: 
                - '16:00'
`

Produced by package command:
`
         CreateRule:
            Interval: 12
            IntervalUnit: HOURS
            Times:
            - 16:00
`
I see the issue when trying to create a custom resource for the AWS Instance Scheduler.

It works in the CodeBuild image `aws/codebuild/standard:4.0` which uses:
`aws-cli/1.25.1 Python/3.8.13 Linux/4.14.275-207.503.amzn2.x86_64 exec-env/AWS_ECS_EC2 botocore/1.27.1`

It does not work with CodeBuild image `aws/codebuild/standard:5.0` which uses:
`aws-cli/2.7.5 Python/3.9.11 Linux/4.14.275-207.503.amzn2.x86_64 exec-env/AWS_ECS_EC2 exe/x86_64.ubuntu.20 prompt/off`

The input looks like this:
```
AWSTemplateFormatVersion: '2010-09-09'
Description: AWS Instance Schedule Definition Stack
Parameters:
  ServiceToken:
    Type: String
Resources:
  EC2Schedule:
    Type: Custom::ServiceInstanceSchedule
    Properties:
      Name: EC2Schedule
      Description: Ec2 Uptime
      NoStackPrefix: 'True'
      Enforced: 'False'
      Hibernate: 'False'
      Timezone: Europe/Berlin
      ServiceToken: !Ref 'ServiceToken'
      Periods:
        - Description: End on weekdays
          EndTime: '23:00'
          WeekDays: Mon-Fri
        - Description: Core Working hours Thursday
          BeginTime: '08:00'
          EndTime: '18:00'
          WeekDays: Thu
        - Description: Weekend Maintenance Window
          BeginTime: '00:01'
          EndTime: '01:30'
          WeekDays: Sun
```

Output looks like this:
```
AWSTemplateFormatVersion: '2010-09-09'
Description: AWS Instance Schedule Definition Stack
Parameters:
  ServiceToken:
    Type: String
Resources:
  EC2Schedule:
    Type: Custom::ServiceInstanceSchedule
    Properties:
      Name: EC2Schedule
      Description: Ec2 Uptime
      NoStackPrefix: 'True'
      Timezone: Europe/Amsterdam
      ServiceToken:
        Ref: ServiceToken
      Periods:
      - Description: End on weekdays
        EndTime: 23:00
        WeekDays: Mon-Fri
      - Description: Core Working hours Thursday
        BeginTime: 08:00
        EndTime: 18:00
        WeekDays: Thu
      - Description: Weekend Maintenance Window
        BeginTime: 00:01
        EndTime: 01:30
        WeekDays: Sun
```


Resolving patch:
diff --git a/awscli/customizations/cloudformation/yamlhelper.py b/awscli/customizations/cloudformation/yamlhelper.py
--- a/awscli/customizations/cloudformation/yamlhelper.py
+++ b/awscli/customizations/cloudformation/yamlhelper.py
@@ -63,21 +63,17 @@ def _dict_representer(dumper, data):
     return dumper.represent_dict(data.items())
 
 
-def _add_yaml_1_1_boolean_resolvers(resolver_cls):
-    # CloudFormation treats unquoted values that are YAML 1.1 native
-    # booleans as booleans, rather than strings. In YAML 1.2, the only
-    # boolean values are "true" and "false" so values such as "yes" and "no"
-    # when loaded as strings are not quoted when dumped. This logic ensures
-    # that we dump these values with quotes so that CloudFormation treats
-    # these values as strings and not booleans.
-    boolean_regex = re.compile(
-        '^(?:yes|Yes|YES|no|No|NO'
-        '|true|True|TRUE|false|False|FALSE'
-        '|on|On|ON|off|Off|OFF)$'
-    )
-    boolean_first_chars = list(u'yYnNtTfFoO')
-    resolver_cls.add_implicit_resolver(
-        'tag:yaml.org,2002:bool', boolean_regex, boolean_first_chars)
+def _str_representer(dumper, data):
+    # ruamel removes quotes from values that are unambiguously strings.
+    # Values like 0888888 can only be a string because integers can't
+    # have leading 0s and octals can't have the digits 8 and 9.
+    # However, CloudFormation treats these nonoctal values as integers
+    # and removes the leading 0s. This logic ensures that nonoctal
+    # values are quoted when dumped.
+    style = None
+    if re.match('^0[0-9]*[89][0-9]*$', data):
+        style = "'"
+    return dumper.represent_scalar('tag:yaml.org,2002:str', data, style=style)
 
 
 def yaml_dump(dict_to_dump):
@@ -88,10 +84,11 @@ def yaml_dump(dict_to_dump):
     """
 
     yaml = ruamel.yaml.YAML(typ="safe", pure=True)
+    yaml.version = (1, 1)
     yaml.default_flow_style = False
     yaml.Representer = FlattenAliasRepresenter
-    _add_yaml_1_1_boolean_resolvers(yaml.Resolver)
     yaml.Representer.add_representer(OrderedDict, _dict_representer)
+    yaml.Representer.add_representer(str, _str_representer)
 
     return dump_yaml_to_str(yaml, dict_to_dump)
 
@@ -111,12 +108,12 @@ def yaml_parse(yamlstr):
         return json.loads(yamlstr, object_pairs_hook=OrderedDict)
     except ValueError:
         yaml = ruamel.yaml.YAML(typ="safe", pure=True)
+        yaml.version = (1, 1)
         yaml.Constructor.add_constructor(
             ruamel.yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG,
             _dict_constructor)
         yaml.Constructor.add_multi_constructor(
             "!", intrinsics_multi_constructor)
-        _add_yaml_1_1_boolean_resolvers(yaml.Resolver)
 
         return yaml.load(yamlstr)
 
diff --git a/awscli/testutils.py b/awscli/testutils.py
--- a/awscli/testutils.py
+++ b/awscli/testutils.py
@@ -34,6 +34,7 @@
 from pprint import pformat
 from subprocess import Popen, PIPE
 import unittest
+import re
 
 from awscli.compat import StringIO
 
@@ -58,6 +59,7 @@
 import awscli.clidriver
 from awscli.plugin import load_plugins
 from awscli.clidriver import CLIDriver
+from awscli.customizations.cloudformation.yamlhelper import yaml_dump
 
 
 _LOADER = botocore.loaders.Loader()
@@ -1012,3 +1014,8 @@ def wait(self, check, *args, **kwargs):
     def _fail_message(self, attempts, successes):
         format_args = (attempts, successes)
         return 'Failed after %s attempts, only had %s successes' % format_args
+
+
+def yaml_dump_without_header(dict_to_dump):
+    dumped_yaml = yaml_dump(dict_to_dump)
+    return re.sub('%YAML 1.1\n---(\n| )', '', dumped_yaml)


Test patch:
diff --git a/tests/functional/cloudformation/test_package.py b/tests/functional/cloudformation/test_package.py
--- a/tests/functional/cloudformation/test_package.py
+++ b/tests/functional/cloudformation/test_package.py
@@ -23,7 +23,7 @@
 from awscli.customizations.cloudformation.artifact_exporter import make_zip
 from awscli.customizations.cloudformation.yamlhelper import yaml_dump
 from awscli.customizations.cloudformation.artifact_exporter import Template
-from awscli.testutils import skip_if_windows
+from awscli.testutils import yaml_dump_without_header, skip_if_windows
 
 
 class TestPackageZipFiles(TestCase):
@@ -87,7 +87,7 @@ def _generate_template_cases():
 def test_known_templates(input_template, output_template):
     template = Template(input_template, os.getcwd(), None)
     exported = template.export()
-    result = yaml_dump(exported)
+    result = yaml_dump_without_header(exported)
     expected = open(output_template, 'r').read()
 
     assert result == expected, (
diff --git a/tests/unit/customizations/cloudformation/test_yamlhelper.py b/tests/unit/customizations/cloudformation/test_yamlhelper.py
--- a/tests/unit/customizations/cloudformation/test_yamlhelper.py
+++ b/tests/unit/customizations/cloudformation/test_yamlhelper.py
@@ -20,6 +20,7 @@
 from awscli.customizations.cloudformation.deployer import Deployer
 from awscli.customizations.cloudformation.yamlhelper import yaml_parse, yaml_dump
 from tests.unit.customizations.cloudformation import BaseYAMLTest
+from awscli.testutils import yaml_dump_without_header
 
 
 class TestYaml(BaseYAMLTest):
@@ -145,7 +146,7 @@ def test_parse_yaml_preserve_elements_order(self):
         ])
         self.assertEqual(expected_dict, output_dict)
 
-        output_template = yaml_dump(output_dict)
+        output_template = yaml_dump_without_header(output_dict)
         self.assertEqual(input_template, output_template)
 
     def test_yaml_merge_tag(self):
@@ -182,7 +183,7 @@ def test_unroll_yaml_anchors(self):
             '      Foo: bar\n'
             '      Spam: eggs\n'
         )
-        actual = yaml_dump(template)
+        actual = yaml_dump_without_header(template)
         self.assertEqual(actual, expected)
 
     def test_yaml_dump_quotes_boolean_strings(self):
@@ -193,4 +194,26 @@ def test_yaml_dump_quotes_boolean_strings(self):
         ]
         for bool_as_string in bools_as_strings:
             self.assertEqual(
-                yaml_dump(bool_as_string), "'%s'\n" % bool_as_string)
+                yaml_dump_without_header(bool_as_string),
+                "'%s'\n" % bool_as_string)
+
+    def test_yaml_dump_quotes_octal_strings(self):
+        octals_as_strings = [
+            '01111111', '02222222', '03333333',
+            '04444444', '05555555', '06666666',
+            '07777777', '08888888', '09999999'
+        ]
+        for octal_as_string in octals_as_strings:
+            self.assertEqual(
+                yaml_dump_without_header(octal_as_string),
+                "'%s'\n" % octal_as_string)
+
+    def test_yaml_dump_include_1_1_header(self):
+        template = {"Resources": {"Foo": "Bar"}}
+        expected = (
+            '%YAML 1.1\n---\n'
+            'Resources:\n'
+            '  Foo: Bar\n'
+        )
+        actual = yaml_dump(template)
+        self.assertEqual(actual, expected)