Repository: getmoto/moto

Issue:
Different ECS behaviors for task-definition with boto3 versus cloudformation
Fetching an cloudformation created ECS taskDefinition using moto ecs client includes empty ContainerDefinition.


I have one ECS TaskDefinition created through cloudformation that completes successfully. However, when I use moto ecs client to fetch this task definition, I see empty body for "ContainerDefinition" that I did set in my cloudformation template. I see that internal moto ECS models script does have a proper list retrieved for "ContainerDefinition" but the list is lost somewhere during parsing json response inside this models.py script of ecs. ( I noticed that parser expected camelCase property name in the ContainerDefinition json body whereas it finds PascalCase attributes naming and hence move forward without preparing ContainerDefinition list)

I digged deeper and found out that the ContainerDefinition resource properties has PascalCase namings whereas the parser models.py expected camelCase and hence ends up  pushing empty list for an earlier available ContainerDefinition. 

I then used an actual boto3 call to fetch a physically available TaskDefinition in my account (this TaskDefinition was created through boto3 cloudformation). 

The details of ContainerDefinition json retrieved using moto ecs client and actual boto3 client are provided below:

ContainerDefinition list below when fetching ECS TaskDefinition through actual boto3 ecs client: (camelCase attribute names)

    [{
	    'essential': True,
	    'image': '123456789012.dkr.ecr.us-east-1.amazonaws.com/domain/my-image',
	    'environment': [{
		'name': 'MyEnvironment',
		'value': 'Devl'
	    }, {
		'name': 'serverport',
		'value': '80'
	    }],
	    'healthCheck': {
		'command': ['CMD-SHELL', 'curl -f http://localhost/health || exit 1'],
		'interval': 30,
		'retries': 3,
		'startPeriod': 300
	    },
	    'logConfiguration': {
		'logDriver': 'awslogs',
		'options': {
			'awslogs-group': 'myapp-log-devl',
			'awslogs-stream-prefix': 'Service',
			'awslogs-region': 'us-west-1'
		}
	    },
	    'name': 'myapp-task-def-devl',
	    'portMappings': [{
		'containerPort': '80',
		'protocol': 'tcp'
	    }]
    }] 



ContainerDefinition list below when fetching ECS TaskDefinition through moto ecs client: (PascalCase attribute names)

        [{
	        'Essential': True,
	        'Image': '123456789012.dkr.ecr.us-east-1.amazonaws.com/domain/my-image',
	        'Environment': [{
		    'Name': 'MyEnvironment',
		    'Value': 'Devl'
	        }, {
		    'Name': 'serverport',
		    'Value': '80'
	        }],
	        'HealthCheck': {
		    'Command': ['CMD-SHELL', 'curl -f http://localhost/health || exit 1'],
		    'Interval': 30,
		    'Retries': 3,
		    'StartPeriod': 300
	        },
	        'LogConfiguration': {
		    'LogDriver': 'awslogs',
		    'Options': {
		  	  'awslogs-group': 'myapp-log-devl',
			  'awslogs-stream-prefix': 'Service',
			  'awslogs-region': 'us-west-1'
		    }
	        },
	        'Name': 'myapp-task-def-devl',
	        'PortMappings': [{
		    'ContainerPort': '80',
		    'Protocol': 'tcp'
	        }]
    }] 


Finally, I ran another test and created an ECS TaskDefinition through moto ecs client (no cloudformation) and verified that fetching TaskDefinition created that way returns me ContainerDefinition as expected and attributes have camelCase namings. 

Does moto cloudformation not support ECS TaskDefinition? Do I need to create TaskDefinition using explicit moto ECS client. 

P.S.: I am already using ECS moto client for creation of TaskSets as its not currently supported through moto cloudformation.

My environment:
      
    boto3 Version: 1.14.22
    botocore Version: 1.17.27
    moto Version: 1.3.15.dev979.   ( I am using this version in order to be able to run ecs.create_task_set()



Discussion:
Hi @pinkjosh, thanks for raising this. You're on the money with the camelCase/PascalCase observation. We do store the containerDefinition internally, but we should convert it's properties to lowercase before sending this back to the client.

Marking it as a bug.
@bblommers Requesting your comments if there is an update on this one.

Resolving patch:
diff --git a/moto/core/utils.py b/moto/core/utils.py
--- a/moto/core/utils.py
+++ b/moto/core/utils.py
@@ -57,6 +57,11 @@ def underscores_to_camelcase(argument):
     return result
 
 
+def pascal_to_camelcase(argument):
+    """Converts a PascalCase param to the camelCase equivalent"""
+    return argument[0].lower() + argument[1:]
+
+
 def method_names_from_class(clazz):
     # On Python 2, methods are different from functions, and the `inspect`
     # predicates distinguish between them. On Python 3, methods are just
@@ -367,3 +372,30 @@ def tags_from_cloudformation_tags_list(tags_list):
         tags[key] = value
 
     return tags
+
+
+def remap_nested_keys(root, key_transform):
+    """This remap ("recursive map") function is used to traverse and
+    transform the dictionary keys of arbitrarily nested structures.
+    List comprehensions do not recurse, making it tedious to apply
+    transforms to all keys in a tree-like structure.
+
+    A common issue for `moto` is changing the casing of dict keys:
+
+    >>> remap_nested_keys({'KeyName': 'Value'}, camelcase_to_underscores)
+    {'key_name': 'Value'}
+
+    Args:
+        root: The target data to traverse. Supports iterables like
+            :class:`list`, :class:`tuple`, and :class:`dict`.
+        key_transform (callable): This function is called on every
+            dictionary key found in *root*.
+    """
+    if isinstance(root, (list, tuple)):
+        return [remap_nested_keys(item, key_transform) for item in root]
+    if isinstance(root, dict):
+        return {
+            key_transform(k): remap_nested_keys(v, key_transform)
+            for k, v in six.iteritems(root)
+        }
+    return root
diff --git a/moto/ecs/models.py b/moto/ecs/models.py
--- a/moto/ecs/models.py
+++ b/moto/ecs/models.py
@@ -11,7 +11,7 @@
 
 from moto.core import BaseBackend, BaseModel, CloudFormationModel
 from moto.core.exceptions import JsonRESTError
-from moto.core.utils import unix_time
+from moto.core.utils import unix_time, pascal_to_camelcase, remap_nested_keys
 from moto.ec2 import ec2_backends
 from .exceptions import (
     ServiceNotFoundException,
@@ -174,8 +174,10 @@ def create_from_cloudformation_json(
         family = properties.get(
             "Family", "task-definition-{0}".format(int(random() * 10 ** 6))
         )
-        container_definitions = properties["ContainerDefinitions"]
-        volumes = properties.get("Volumes")
+        container_definitions = remap_nested_keys(
+            properties.get("ContainerDefinitions", []), pascal_to_camelcase
+        )
+        volumes = remap_nested_keys(properties.get("Volumes", []), pascal_to_camelcase)
 
         ecs_backend = ecs_backends[region_name]
         return ecs_backend.register_task_definition(


Test patch:
diff --git a/tests/test_ecs/test_ecs_cloudformation.py b/tests/test_ecs/test_ecs_cloudformation.py
--- a/tests/test_ecs/test_ecs_cloudformation.py
+++ b/tests/test_ecs/test_ecs_cloudformation.py
@@ -2,6 +2,8 @@
 import json
 from copy import deepcopy
 from moto import mock_cloudformation, mock_ecs
+from moto.core.utils import pascal_to_camelcase, remap_nested_keys
+import sure  # noqa
 
 
 @mock_ecs
@@ -231,9 +233,16 @@ def test_create_task_definition_through_cloudformation():
                             "Cpu": "200",
                             "Memory": "500",
                             "Essential": "true",
+                            "PortMappings": [
+                                {
+                                    "ContainerPort": 123,
+                                    "HostPort": 123,
+                                    "Protocol": "tcp",
+                                },
+                            ],
                         }
                     ],
-                    "Volumes": [],
+                    "Volumes": [{"Name": "ecs-vol"}],
                 },
             }
         },
@@ -252,3 +261,14 @@ def test_create_task_definition_through_cloudformation():
         StackName=stack_name, LogicalResourceId="testTaskDefinition"
     )["StackResourceDetail"]
     task_definition_details["PhysicalResourceId"].should.equal(task_definition_arn)
+
+    task_definition = ecs_conn.describe_task_definition(
+        taskDefinition=task_definition_arn
+    ).get("taskDefinition")
+    expected_properties = remap_nested_keys(
+        template["Resources"]["testTaskDefinition"]["Properties"], pascal_to_camelcase
+    )
+    task_definition["volumes"].should.equal(expected_properties["volumes"])
+    task_definition["containerDefinitions"].should.equal(
+        expected_properties["containerDefinitions"]
+    )