Repository: aws/aws-cli

Issue:
Add `cloudformation package` support for AWS::Glue::Job resource type
Instead of having to upload Glue scripts to S3 manually, it would be nice if it could be done with `aws cloudformation package` command the same way it's done for `AWS::CloudFormation::Stack` or `AWS::Lambda::Function` resource types:

```yaml
Resources:
  MyGlueJob:
    Type: AWS::Glue::Job
    Properties:
      Name: MyGlueJob
      Command:
        ScriptLocation: ./gluescript.scala
        Name: glueetl
        ...
```


Discussion:
@ilyasotkov Thanks for reaching out.  This seems like it would be very helpful.  I have changed the label to feature request. @sanathkr what are your thoughts on this idea?
What are the current workarounds ?
Kill me but i cant find how to provide inline code for  glue in cloudformation template. I refuse to believe that the only option is to actually create bucket manually, then upload the script, and then hopping that i get the path right paste it into the cloudformation template. 
We should implement this. It should be a few lines of code change and some tests change. Could be a good starter project for someone. PRs welcome!
My mates need this feature and I gave it a 5 minutes investigation, which I'll share here.

The relevant bit is [artifact_exporter.py](https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudformation/artifact_exporter.py). I thought about creating a new subclass of `Resource` just like `ServerlessFunctionResource` but then I realised that the properties of all supported resources are top level, while with glue script we need to one level down: `AWS::Glue::Job.JobCommand.ScriptLocation`. It might be trivial but I don't think this is currently supported by the [`Resource`](https://github.com/aws/aws-cli/blob/3251a4b633ee2e09b11463871fa27e0139011864/awscli/customizations/cloudformation/artifact_exporter.py#L216) class.

Happy to be proven wrong of course!
@sanathkr and @andreabedini - Thanks for your feedback.  I added the 'needs-contributors' label. 
This came up at with some teams at my company today.  I'm looking into implementing this.  Here's what I'm proposing as an implementation.  Please provide any feedback.  I'll open a PR once this is implemented.

As @andreabedini mentioned, the relevant functionality is in [artifact_exporter.py](customizations/cloudformation/artifact_exporter.py).  And the complexity is introduced by the ScriptLocation being nested under the Location property of the AWS::Glue::Job resource.

Add a new GlueCommandJobCommandScriptLocationResource.
```
class GlueCommandJobCommandScriptLocationResource(Resource):
    RESOURCE_TYPE = "AWS::Glue::Job"
    # Note the PROPERTY_NAME includes a '.' implying it's nested.
    PROPERTY_NAME = "Command.ScriptLocation"
```

In order to support the nested property (ie Command.ScriptLocation), we will need to replace all of the direct resource_dict accessors, with 2 new get_nested_property_value and set_nested_property_value methods that will split the PROPERTY_NAME on '.' and traverse the resource_dict to find the appropriate object.

```
def get_nested_property_value(resource_dict, property_name):
    """
    Searches the resource_dict for nested properties by splitting the property_name
    on '.'

    :param resource_dict:   Dictionary containing resource definition
    :param property_name:   Property name of CloudFormation resource where this
                            local path is present

    :return:                Value of the property
    """

    # Support nested properties by allowing '.' in the PROPERTY_NAME
    if '.' in property_name:
        sub_property_names = property_name.split('.')
        property_value = resource_dict.get(sub_property_names[0], None)
        for sub_property_name in sub_property_names[1:]:
            if property_value:
                property_value = property_value.get(sub_property_name, None)
    else:
        property_value = resource_dict.get(property_name, None)

    return property_value


def set_nested_property_value(resource_dict, property_name, new_value):
    """
    Searches the resource_dict for nested properties by splitting the property_name
    on '.' and sets the nested property to new_value.


    :param resource_dict:   Dictionary containing resource definition
    :param property_name:   Property name of CloudFormation resource where this
                            local path is present
    :param new_value:       The new value for the property
    :raise:                 KeyError if a property isn't found in the dictionary
    """
    # Support nested properties by allowing '.' in the PROPERTY_NAME
    # Assumes that the property exists in the dictionary
    if '.' in property_name:
        sub_property_names = property_name.split('.')
        property_dict = resource_dict[sub_property_names[0]]
        for sub_property_name in sub_property_names[1:-1]:
            property_dict = property_dict[sub_property_name]
        property_dict[sub_property_names[-1]] = new_value
    else:
        resource_dict[property_name] = new_value
```

Resolving patch:
diff --git a/awscli/customizations/cloudformation/artifact_exporter.py b/awscli/customizations/cloudformation/artifact_exporter.py
--- a/awscli/customizations/cloudformation/artifact_exporter.py
+++ b/awscli/customizations/cloudformation/artifact_exporter.py
@@ -19,12 +19,15 @@
 import uuid
 import shutil
 from awscli.compat import six
+from botocore.utils import set_value_from_jmespath
 
 from awscli.compat import urlparse
 from contextlib import contextmanager
 from awscli.customizations.cloudformation import exceptions
 from awscli.customizations.cloudformation.yamlhelper import yaml_dump, \
     yaml_parse
+import jmespath
+
 
 LOG = logging.getLogger(__name__)
 
@@ -120,7 +123,7 @@ def upload_local_artifacts(resource_id, resource_dict, property_name,
     :raise:                 ValueError if path is not a S3 URL or a local path
     """
 
-    local_path = resource_dict.get(property_name, None)
+    local_path = jmespath.search(property_name, resource_dict)
 
     if local_path is None:
         # Build the root directory and upload to S3
@@ -232,7 +235,7 @@ def export(self, resource_id, resource_dict, parent_dir):
         if resource_dict is None:
             return
 
-        property_value = resource_dict.get(self.PROPERTY_NAME, None)
+        property_value = jmespath.search(self.PROPERTY_NAME, resource_dict)
 
         if not property_value and not self.PACKAGE_NULL_PROPERTY:
             return
@@ -248,7 +251,7 @@ def export(self, resource_id, resource_dict, parent_dir):
         if is_local_file(property_value) and not \
                 is_zip_file(property_value) and self.FORCE_ZIP:
             temp_dir = copy_to_temp_dir(property_value)
-            resource_dict[self.PROPERTY_NAME] = temp_dir
+            set_value_from_jmespath(resource_dict, self.PROPERTY_NAME, temp_dir)
 
         try:
             self.do_export(resource_id, resource_dict, parent_dir)
@@ -269,10 +272,10 @@ def do_export(self, resource_id, resource_dict, parent_dir):
         Default export action is to upload artifacts and set the property to
         S3 URL of the uploaded object
         """
-        resource_dict[self.PROPERTY_NAME] = \
-            upload_local_artifacts(resource_id, resource_dict,
+        uploaded_url = upload_local_artifacts(resource_id, resource_dict,
                                    self.PROPERTY_NAME,
                                    parent_dir, self.uploader)
+        set_value_from_jmespath(resource_dict, self.PROPERTY_NAME, uploaded_url)
 
 
 class ResourceWithS3UrlDict(Resource):
@@ -299,11 +302,12 @@ def do_export(self, resource_id, resource_dict, parent_dir):
                                    self.PROPERTY_NAME,
                                    parent_dir, self.uploader)
 
-        resource_dict[self.PROPERTY_NAME] = parse_s3_url(
+        parsed_url = parse_s3_url(
                 artifact_s3_url,
                 bucket_name_property=self.BUCKET_NAME_PROPERTY,
                 object_key_property=self.OBJECT_KEY_PROPERTY,
                 version_property=self.VERSION_PROPERTY)
+        set_value_from_jmespath(resource_dict, self.PROPERTY_NAME, parsed_url)
 
 
 class ServerlessFunctionResource(Resource):
@@ -459,8 +463,9 @@ def do_export(self, resource_id, resource_dict, parent_dir):
 
             # TemplateUrl property requires S3 URL to be in path-style format
             parts = parse_s3_url(url, version_property="Version")
-            resource_dict[self.PROPERTY_NAME] = self.uploader.to_path_style_s3_url(
+            s3_path_url = self.uploader.to_path_style_s3_url(
                     parts["Key"], parts.get("Version", None))
+            set_value_from_jmespath(resource_dict, self.PROPERTY_NAME, s3_path_url)
 
 
 class ServerlessApplicationResource(CloudFormationStackResource):
@@ -472,6 +477,16 @@ class ServerlessApplicationResource(CloudFormationStackResource):
     PROPERTY_NAME = "Location"
 
 
+
+class GlueJobCommandScriptLocationResource(Resource):
+    """
+    Represents Glue::Job resource.
+    """
+    RESOURCE_TYPE = "AWS::Glue::Job"
+    # Note the PROPERTY_NAME includes a '.' implying it's nested.
+    PROPERTY_NAME = "Command.ScriptLocation"
+
+
 RESOURCES_EXPORT_LIST = [
     ServerlessFunctionResource,
     ServerlessApiResource,
@@ -487,6 +502,7 @@ class ServerlessApplicationResource(CloudFormationStackResource):
     ServerlessApplicationResource,
     ServerlessLayerVersionResource,
     LambdaLayerVersionResource,
+    GlueJobCommandScriptLocationResource,
 ]
 
 METADATA_EXPORT_LIST = [


Test patch:
diff --git a/tests/unit/customizations/cloudformation/test_artifact_exporter.py b/tests/unit/customizations/cloudformation/test_artifact_exporter.py
--- a/tests/unit/customizations/cloudformation/test_artifact_exporter.py
+++ b/tests/unit/customizations/cloudformation/test_artifact_exporter.py
@@ -26,7 +26,8 @@
     AppSyncResolverRequestTemplateResource, \
     AppSyncResolverResponseTemplateResource, \
     AppSyncFunctionConfigurationRequestTemplateResource, \
-    AppSyncFunctionConfigurationResponseTemplateResource
+    AppSyncFunctionConfigurationResponseTemplateResource, \
+    GlueJobCommandScriptLocationResource
 
 
 def test_is_s3_url():
@@ -143,6 +144,12 @@ def test_all_resources_export():
         {
             "class": ServerlessRepoApplicationLicense,
             "expected_result": uploaded_s3_url
+        },
+        {
+            "class": GlueJobCommandScriptLocationResource,
+            "expected_result": {
+                    "ScriptLocation": uploaded_s3_url 
+            }
         }
     ]
 
@@ -162,9 +169,22 @@ def _helper_verify_export_resources(
     upload_local_artifacts_mock.reset_mock()
 
     resource_id = "id"
-    resource_dict = {
-        test_class.PROPERTY_NAME: "foo"
-    }
+
+    if '.' in test_class.PROPERTY_NAME:
+        reversed_property_names = test_class.PROPERTY_NAME.split('.')
+        reversed_property_names.reverse()
+        property_dict = {
+            reversed_property_names[0]: "foo"
+        }
+        for sub_property_name in reversed_property_names[1:]:
+            property_dict = {
+                sub_property_name: property_dict
+            }
+        resource_dict = property_dict
+    else:
+        resource_dict = {
+            test_class.PROPERTY_NAME: "foo"
+        }
     parent_dir = "dir"
 
     upload_local_artifacts_mock.return_value = uploaded_s3_url
@@ -178,7 +198,11 @@ def _helper_verify_export_resources(
                                                         test_class.PROPERTY_NAME,
                                                         parent_dir,
                                                         s3_uploader_mock)
-    result = resource_dict[test_class.PROPERTY_NAME]
+    if '.' in test_class.PROPERTY_NAME:
+        top_level_property_name = test_class.PROPERTY_NAME.split('.')[0]
+        result = resource_dict[top_level_property_name]
+    else:
+        result = resource_dict[test_class.PROPERTY_NAME]
     assert_equal(result, expected_result)
 
 
@@ -298,7 +322,6 @@ def test_upload_local_artifacts_local_file_abs_path(self, zip_and_upload_mock):
             self.s3_uploader_mock.upload_with_dedup.assert_called_with(artifact_path)
             zip_and_upload_mock.assert_not_called()
 
-
     @patch("awscli.customizations.cloudformation.artifact_exporter.zip_and_upload")
     def test_upload_local_artifacts_local_folder(self, zip_and_upload_mock):
         property_name = "property"
@@ -325,8 +348,6 @@ def test_upload_local_artifacts_local_folder(self, zip_and_upload_mock):
             zip_and_upload_mock.assert_called_once_with(absolute_artifact_path,
                                                         mock.ANY)
 
-
-
     @patch("awscli.customizations.cloudformation.artifact_exporter.zip_and_upload")
     def test_upload_local_artifacts_no_path(self, zip_and_upload_mock):
         property_name = "property"
@@ -397,7 +418,6 @@ def test_upload_local_artifacts_invalid_value(self, zip_and_upload_mock):
         zip_and_upload_mock.assert_not_called()
         self.s3_uploader_mock.upload_with_dedup.assert_not_called()
 
-
     @patch("awscli.customizations.cloudformation.artifact_exporter.make_zip")
     def test_zip_folder(self, make_zip_mock):
         zip_file_name = "name.zip"
@@ -512,7 +532,6 @@ class MockResource(Resource):
         is_zipfile_mock.assert_called_once_with(original_path)
         assert_equal(resource_dict[resource.PROPERTY_NAME], s3_url)
 
-
     @patch("shutil.rmtree")
     @patch("zipfile.is_zipfile")
     @patch("awscli.customizations.cloudformation.artifact_exporter.copy_to_temp_dir")
@@ -547,7 +566,6 @@ class MockResourceNoForceZip(Resource):
         is_zipfile_mock.assert_called_once_with(original_path)
         assert_equal(resource_dict[resource.PROPERTY_NAME], s3_url)
 
-
     @patch("awscli.customizations.cloudformation.artifact_exporter.upload_local_artifacts")
     def test_resource_empty_property_value(self, upload_local_artifacts_mock):
         # Property value is empty
@@ -633,7 +651,6 @@ class MockResource(Resource):
         with self.assertRaises(exceptions.ExportFailedError):
             resource.export(resource_id, resource_dict, parent_dir)
 
-
     @patch("awscli.customizations.cloudformation.artifact_exporter.upload_local_artifacts")
     def test_resource_with_s3_url_dict(self, upload_local_artifacts_mock):
         """
@@ -674,7 +691,6 @@ class MockResource(ResourceWithS3UrlDict):
             "v": "SomeVersionNumber"
         })
 
-
     @patch("awscli.customizations.cloudformation.artifact_exporter.Template")
     def test_export_cloudformation_stack(self, TemplateMock):
         stack_resource = CloudFormationStackResource(self.s3_uploader_mock)
@@ -1056,7 +1072,6 @@ def test_include_transform_export_handler_non_include_transform(self, is_local_f
         self.s3_uploader_mock.upload_with_dedup.assert_not_called()
         self.assertEquals(handler_output, {"Name": "AWS::OtherTransform", "Parameters": {"Location": "foo.yaml"}})
 
-
     def test_template_export_path_be_folder(self):
 
         template_path = "/path/foo"
@@ -1110,7 +1125,6 @@ def test_copy_to_temp_dir(self, mkdtemp_mock, copyfile_mock):
         self.assertEqual(returned_dir, temp_dir)
         copyfile_mock.assert_called_once_with(filename, temp_dir + filename)
 
-
     @contextmanager
     def make_temp_dir(self):
         filename = tempfile.mkdtemp()