Repository: aws/aws-cli

Issue:
cloudformation package should preserve comments and order of parameters
Given the `template.yaml`:
```yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  # This is a comment
  VotesTable:
    Type: AWS::Serverless::SimpleTable
  VoteSpacesTabs:
    # this is another comment
    Type: AWS::Serverless::Function
    Properties:
      Handler: lambda_function.lambda_handler
      Policies: AmazonDynamoDBFullAccess
      Runtime: python3.6
      Environment:
        Variables:
          TABLE_NAME:
            Ref: VotesTable
      Events:
        Vote:
          Properties:
            Method: post
            Path: /
          Type: Api
      Timeout: 10
```

And the command:
`aws cloudformation package --template-file `pwd`/template.yaml --s3-bucket ranman-code --output-template-file package.yaml`

package.yaml should have:
```yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  # This is a comment
  VotesTable:
    Type: AWS::Serverless::SimpleTable
  VoteSpacesTabs:
    # this is another comment
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: s3://ranman-code/97402a11ac040008e15358ccc872f799
      Handler: lambda_function.lambda_handler
      Policies: AmazonDynamoDBFullAccess
      Runtime: python3.6
      Environment:
        Variables:
          TABLE_NAME:
            Ref: VotesTable
      Events:
        Vote:
          Properties:
            Method: post
            Path: /
          Type: Api
      Timeout: 10
```

Instead it has:

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  VoteSpacesTabs:
    Properties:
      CodeUri: s3://ranman-code/97402a11ac040008e15358ccc872f799
      Environment:
        Variables:
          TABLE_NAME:
            Ref: VotesTable
      Events:
        Vote:
          Properties:
            Method: post
            Path: /
          Type: Api
      Handler: lambda_function.lambda_handler
      Policies: AmazonDynamoDBFullAccess
      Runtime: python3.6
      Timeout: 10
    Type: AWS::Serverless::Function
  VotesTable:
    Type: AWS::Serverless::SimpleTable
Transform: AWS::Serverless-2016-10-31
```

```diff
@@ -1,16 +1,8 @@
 AWSTemplateFormatVersion: '2010-09-09'
-Transform: AWS::Serverless-2016-10-31
 Resources:
-  # This is a comment
-  VotesTable:
-    Type: AWS::Serverless::SimpleTable
   VoteSpacesTabs:
-    # this is another comment
-    Type: AWS::Serverless::Function
     Properties:
-      Handler: lambda_function.lambda_handler
-      Policies: AmazonDynamoDBFullAccess
-      Runtime: python3.6
+      CodeUri: s3://ranman-code/97402a11ac040008e15358ccc872f799
       Environment:
         Variables:
           TABLE_NAME:
@@ -21,4 +13,11 @@
             Method: post
             Path: /
           Type: Api
+      Handler: lambda_function.lambda_handler
+      Policies: AmazonDynamoDBFullAccess
+      Runtime: python3.6
       Timeout: 10
\ No newline at end of file
+    Type: AWS::Serverless::Function
+  VotesTable:
+    Type: AWS::Serverless::SimpleTable
+Transform: AWS::Serverless-2016-10-31
```


Discussion:
@sanathkr Any thoughts on this?
This is coming directly from PyYaml. If it can provide us hooks to retain ordering and comments, then yes, we should be able to enable it.
I think using the roundtrip parser from https://pypi.python.org/pypi/ruamel.yaml might solve the issue. http://yaml.readthedocs.io/en/latest/example.html


@ranman That would require switching from `PyYaml` to `ruamel`. We've actually discussed this in the past and aren't sure if the benefits of switching outweigh the possibility for regression or unintended breaking changes. I think it would be best if this can be done within `PyYaml`. It seems that the `PyYaml` devs aren't too keen on have reliable ordering as it goes against the yaml spec.
Good Morning!

We're closing this issue here on GitHub, as part of our migration to [UserVoice](https://aws.uservoice.com/forums/598381-aws-command-line-interface) for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake.  As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.  

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface 

-The AWS SDKs & Tools Team
Based on community feedback, we have decided to return feature requests to GitHub issues.

Resolving patch:
diff --git a/awscli/customizations/cloudformation/yamlhelper.py b/awscli/customizations/cloudformation/yamlhelper.py
--- a/awscli/customizations/cloudformation/yamlhelper.py
+++ b/awscli/customizations/cloudformation/yamlhelper.py
@@ -11,12 +11,14 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
 import ruamel.yaml as yaml
+from ruamel.yaml.comments import CommentedMap
+
 from ruamel.yaml.resolver import ScalarNode, SequenceNode
 from botocore.compat import json
 from botocore.compat import OrderedDict
 
 
-from awscli.compat import six
+from awscli.compat import six, StringIO
 
 
 def intrinsics_multi_constructor(loader, tag_prefix, node):
@@ -51,13 +53,14 @@ def intrinsics_multi_constructor(loader, tag_prefix, node):
 
     else:
         # Value of this node is an mapping (ex: {foo: bar})
-        value = loader.construct_mapping(node)
+        value = CommentedMap()
+        loader.construct_mapping(node, value)
 
     return {cfntag: value}
 
 
 def _dict_representer(dumper, data):
-    return dumper.represent_dict(data.items())
+    return dumper.represent_dict(data)
 
 
 def yaml_dump(dict_to_dump):
@@ -66,12 +69,13 @@ def yaml_dump(dict_to_dump):
     :param dict_to_dump:
     :return:
     """
-    FlattenAliasDumper.add_representer(OrderedDict, _dict_representer)
-    return yaml.dump(
-        dict_to_dump,
-        default_flow_style=False,
-        Dumper=FlattenAliasDumper,
-    )
+    y = yaml.YAML(typ='rt')
+    y.default_flow_style = False
+    y.representer.add_representer(OrderedDict, _dict_representer)
+    y.representer.ignore_aliases = lambda data: True
+    output = StringIO()
+    y.dump(dict_to_dump, output)
+    return output.getvalue()
 
 
 def _dict_constructor(loader, node):
@@ -88,12 +92,7 @@ def yaml_parse(yamlstr):
         # json parser.
         return json.loads(yamlstr, object_pairs_hook=OrderedDict)
     except ValueError:
-        yaml.SafeLoader.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _dict_constructor)
-        yaml.SafeLoader.add_multi_constructor(
+        y = yaml.YAML(typ='rt')
+        y.constructor.add_multi_constructor(
             "!", intrinsics_multi_constructor)
-        return yaml.safe_load(yamlstr)
-
-
-class FlattenAliasDumper(yaml.SafeDumper):
-    def ignore_aliases(self, data):
-        return True
+        return y.load(yamlstr)


Test patch:
diff --git a/tests/functional/ddb/test_put.py b/tests/functional/ddb/test_put.py
--- a/tests/functional/ddb/test_put.py
+++ b/tests/functional/ddb/test_put.py
@@ -25,10 +25,22 @@ def setUp(self):
         super(TestPut, self).setUp()
         self.parsed_response = {}
         self.tempdir = tempfile.mkdtemp()
+        self.original_tag_handlers = yaml.YAML(typ='safe').constructor\
+            .yaml_constructors.copy()
 
     def tearDown(self):
         super(TestPut, self).tearDown()
         shutil.rmtree(self.tempdir)
+        # This line looks wrong, right?  Well... the "yaml_constructors"
+        # is actually a class attribute that's shared across *all* of the
+        # yaml.YAML() instances.  Because the ddb customization registers
+        # a custom handler for the binary and float types, this will break
+        # other code (and tests) that rely on the default behavior.  So
+        # to fix this, we put back the original tag handlers that we saved
+        # in self.original_tag_handlers to ensure we handle binary and
+        # float types correctly.
+        yaml.YAML(typ='safe').constructor.yaml_constructors.update(
+            self.original_tag_handlers)
 
     def assert_yaml_response_equal(self, response, expected):
         with self.assertRaises(ValueError):
diff --git a/tests/unit/customizations/cloudformation/test_yamlhelper.py b/tests/unit/customizations/cloudformation/test_yamlhelper.py
--- a/tests/unit/customizations/cloudformation/test_yamlhelper.py
+++ b/tests/unit/customizations/cloudformation/test_yamlhelper.py
@@ -24,6 +24,8 @@
 
 class TestYaml(unittest.TestCase):
 
+    maxDiff = None
+
     yaml_with_tags = """
     Resource:
         Key1: !Ref Something
@@ -156,19 +158,18 @@ def test_yaml_merge_tag(self):
             <<: *base
         """
         output = yaml_parse(test_yaml)
-        self.assertTrue(isinstance(output, OrderedDict))
         self.assertEqual(output.get('test').get('property'), 'value')
 
     def test_unroll_yaml_anchors(self):
-        properties = {
-            "Foo": "bar",
-            "Spam": "eggs",
-        }
+        properties = OrderedDict([
+            ("Foo", "bar"),
+            ("Spam", "eggs"),
+        ])
         template = {
-            "Resources": {
-                "Resource1": {"Properties": properties},
-                "Resource2": {"Properties": properties}
-            }
+            "Resources": OrderedDict([
+                ("Resource1", {"Properties": properties}),
+                ("Resource2", {"Properties": properties}),
+            ])
         }
 
         expected = (
@@ -184,3 +185,15 @@ def test_unroll_yaml_anchors(self):
         )
         actual = yaml_dump(template)
         self.assertEqual(actual, expected)
+
+    def test_can_roundtrip_comments(self):
+        test_yaml = (
+            'foo:\n # This is a comment'
+            '  # So is this.\n'
+            '  bar: baz\n'
+            '  baz: qux  # Another comment\n'
+            'list: [1, 2, 3] # List comment\n'
+        )
+        parsed = yaml_parse(test_yaml)
+        round_tripped = yaml_dump(parsed)
+        self.assertEqual(test_yaml, round_tripped)