Repository: aws/aws-sam-cli

Issue:
Layers shared in multiple stacks cause build/deploy to fail
<!-- Make sure we don't have an existing Issue that reports the bug you are seeing (both open and closed). 
If you do find an existing Issue, re-open or add a comment to that Issue instead of creating a new one. -->

The issue I am facing seems to be because of a recent update to SAM cli. Since my SAM templates did not change recently and my deploy was running fine up until a few days ago. Possibly a bug introduced in 1.21.x or 1.20.0.

### Description:
<!-- Briefly describe the bug you are facing.-->
`sam build` errors out with following message:
`Error: L is an Invalid Layer Arn.`

_The following steps will help replicate the issue, but please refer to the "debug" output at the bottom. Because I think the underlying problem is exposed there._

I create a nested application in this case. In the parent application's template there are 2 layers defined. I pass the layer ARNs from parent to child template using Parameters. The error origins where the layer ARNs are specified inside a lambda function resource definition.


### Steps to reproduce:
<!-- Provide detailed steps to replicate the bug, including steps from third party tools (CDK, etc.) -->

Parent template:
```
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  AWSAccountId:
    Type: String
    Default: "123"
  IAMRoleName:
    Type: String
    Default: "role"

Resources:
  OneLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: "one-layer"
      ContentUri: ../layers/one/
      CompatibleRuntimes:
        - python3.8
      RetentionPolicy: Delete

  AnotherLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: "another-layer"
      ContentUri: ../layers/another/
      CompatibleRuntimes:
        - python3.8
      RetentionPolicy: Delete

  SomeLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../lambdas/example/
      FunctionName: "SomeLambdaName"
      Handler: test.lambda_handler
      Role: !Ref IAMRole
      Layers:
        - !Ref OneLayer
        - !Ref AnotherLayer

  TestApp:
    Type: AWS::Serverless::Application
    Properties:
      Location: apps/test.yaml
      Parameters:
        IAMRole: !Join ['', ["arn:aws:iam::", !Ref AWSAccountId, ":role/", !Ref IAMRoleName]]
        Layers: !Join [",", [!Ref OneLayer, !Ref AnotherLayer]]
```
Nested App template:
```
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  IAMRole:
    Type: String
  Layers:
    Type: CommaDelimitedList

Resources:
  NestedLambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../lambdas/example/
      FunctionName: "NestedLambdaName"
      Handler: nested.lambda_handler
      Role: !Ref IAMRole
      Layers: !Ref Layers
```


### Observed result:
<!-- Please provide command output with `--debug` flag set. -->
```
Error: L is an Invalid Layer Arn.
```
### Expected result:
<!-- Describe what you expected. -->

```
Build Succeeded

Built Artifacts  ...
.
.
```

### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS/Ubuntu
2. `sam --version`: 1.21.1/1.21.0
3. AWS region: us-east-1

With the `--debug` flag the log statement in output that stands out is:
`Unable to resolve property Layers: {'Fn::Join': [',', [{'Ref': 'OneLayer'}, {'Ref': 'AnotherLayer'}]]}. Leaving as is.`

In my actual template I can see more statements like this for other parameters which use `Join` function in value.




Discussion:
After initial investigation, `sam` requires the property `Layers` to be a list of string. According to the [CFN doc](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html#cfn-lambda-function-layers), `Layers` should be of type `List of String`. But apparently CFN supports comma delimited string as well. `sam build` or `sam deploy` will treat this as invalid Arn, which is a bug. This bug exists before 1.20.0, I can reproduced it in 1.19.0 without nested stacks using the template below:

**template A**
```yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  Layers:
    Type: CommaDelimitedList
    Default: "arn:aws:lambda:::awslayer:AmazonLinux1703,arn:aws:lambda:::awslayer:AmazonLinux1803"

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: function
      Handler: app.lambda_handler
      Runtime: python3.8
      Layers: !Ref Layers
```

### Why this is surfaced now?

before 1.20.0, `sam build` won't go into nested stacks and `sam deploy` simply upload all resources in nested stacks to CFN without too much checks. Since 1.20.0, this behavior changed.

[Update] I renamed the title of this issue to better describe the issue.
@aahung Does `!Ref Layers` _not_ resolve to a list of strings in `sam`, since the type for parameter `Layers` is `CommaDelimitedList`.
> @aahung Does `!Ref Layers` _not_ resolve to a list of strings in `sam`, since the type for parameter `Layers` is `CommaDelimitedList`.

Yes this is one, also there might be other issues behind this. We are currently investigating and working on a fix. Sorry about the inconvenience.

A related question: Does your `NestedLambda` in your example require building? I wonder what was your nested stack workflow was and whether rolling back to not building nested stacks is a solution for you.
Yes, in the actual workflow, the nested stack does require building, AFAIK.
For now, pinning the version to 1.19 works for me. At this point, are you able to vaguely estimate when a fix may be released?

TBH, One of the main reasons I _initially_ chose to have nested stacks is for better management of the template. The template grew to a point where it was getting difficult to manage. I wanted a way to break it up and nested stacks was the only way I could find to logically group resources into smaller, more manageable templates. Maybe there should be some feature which would allow including/importing templates. :)
> Yes, in the actual workflow, the nested stack does require building, AFAIK.

in 1.19.0, functions in nested stacks won't get built by `sam build`. Do you build them separately? (I am trying to understand how your workflow works and what we can do to unblock you)

>  For now, pinning the version to 1.19 works for me. At this point, are you able to vaguely estimate when a fix may be released?

We are actively exploring possible solutions to this problem and treating it as a matter of top priority - we'll push out a patch release as soon as it's ready. We'll update this issue with a link to the pull request as soon as we create it.

Also I want to say your use case is very valuable for our development. Thank you for providing info about your template!
Thank you for taking this up on priority.

> Do you build them separately?

No. I do a `sam build` on the parent and then a `sam deploy` on the parent too.
> Thank you for taking this up on priority.
> 
> > Do you build them separately?
> 
> No. I do a `sam build` on the parent and then a `sam deploy` on the parent too.

ok, if `sam build` is only executed on the parent, before 1.20.0, resources in nested stacks are not built. 

When you run `sam build`, you can see which functions are built in stdout.

```
Building codeuri: ...... functions: [<the function being built>]
```

Resolving patch:
diff --git a/samcli/lib/providers/provider.py b/samcli/lib/providers/provider.py
--- a/samcli/lib/providers/provider.py
+++ b/samcli/lib/providers/provider.py
@@ -120,6 +120,9 @@ class LayerVersion:
 
     LAYER_NAME_DELIMETER = "-"
 
+    _name: Optional[str] = None
+    _version: Optional[int] = None
+
     def __init__(
         self,
         arn: str,
@@ -149,8 +152,6 @@ def __init__(
         self._arn = arn
         self._codeuri = codeuri
         self.is_defined_within_template = bool(codeuri)
-        self._name = LayerVersion._compute_layer_name(self.is_defined_within_template, arn)
-        self._version = LayerVersion._compute_layer_version(self.is_defined_within_template, arn)
         self._build_method = cast(Optional[str], metadata.get("BuildMethod", None))
         self._compatible_runtimes = compatible_runtimes
 
@@ -238,6 +239,11 @@ def name(self) -> str:
         str
             A name of the Layer that is used on the system to uniquely identify the layer
         """
+        # because self.name is only used in local invoke.
+        # here we delay the validation process (in _compute_layer_name) rather than in __init__() to ensure
+        # customers still have a smooth build experience.
+        if not self._name:
+            self._name = LayerVersion._compute_layer_name(self.is_defined_within_template, self.arn)
         return self._name
 
     @property
@@ -250,6 +256,11 @@ def codeuri(self, codeuri: Optional[str]) -> None:
 
     @property
     def version(self) -> Optional[int]:
+        # because self.version is only used in local invoke.
+        # here we delay the validation process (in _compute_layer_name) rather than in __init__() to ensure
+        # customers still have a smooth build experience.
+        if self._version is None:
+            self._version = LayerVersion._compute_layer_version(self.is_defined_within_template, self.arn)
         return self._version
 
     @property
@@ -284,7 +295,10 @@ def get_build_dir(self, build_root_dir: str) -> str:
 
     def __eq__(self, other: object) -> bool:
         if isinstance(other, type(self)):
-            return self.__dict__ == other.__dict__
+            # self._name and self._version are generated from self._arn, and they are initialized as None
+            # and their values are assigned at runtime. Here we exclude them from comparison
+            overrides = {"_name": None, "_version": None}
+            return {**self.__dict__, **overrides} == {**other.__dict__, **overrides}
         return False
 
 
diff --git a/samcli/lib/providers/sam_function_provider.py b/samcli/lib/providers/sam_function_provider.py
--- a/samcli/lib/providers/sam_function_provider.py
+++ b/samcli/lib/providers/sam_function_provider.py
@@ -421,6 +421,12 @@ def _parse_layer_info(
                         stack_path=stack.stack_path,
                     )
                 )
+            else:
+                LOG.debug(
+                    'layer "%s" is not recognizable, '
+                    "it might be using intrinsic functions that we don't support yet. Skipping.",
+                    str(layer),
+                )
 
         return layers
 


Test patch:
diff --git a/tests/integration/buildcmd/build_integ_base.py b/tests/integration/buildcmd/build_integ_base.py
--- a/tests/integration/buildcmd/build_integ_base.py
+++ b/tests/integration/buildcmd/build_integ_base.py
@@ -304,3 +304,75 @@ def _verify_process_code_and_output(self, command_result, function_full_paths):
     def _verify_invoke_built_functions(self, template_path, overrides, function_and_expected):
         for function_identifier, expected in function_and_expected:
             self._verify_invoke_built_function(template_path, function_identifier, overrides, expected)
+
+
+class IntrinsicIntegBase(BuildIntegBase):
+    """
+    Currently sam-cli does not have great support for intrinsic functions,
+    in this kind of integ tests, there are functions that are buildable but not invocable.
+    """
+
+    def _verify_build(self, function_full_paths, layer_full_path, stack_paths, command_result):
+        """
+        Verify resources have their build artifact folders, stack has their own template.yaml, and command succeeds.
+        """
+        self._verify_process_code_and_output(command_result, function_full_paths, layer_full_path)
+        for function_full_path in function_full_paths:
+            self._verify_build_artifact(self.default_build_dir, function_full_path)
+        for stack_path in stack_paths:
+            self._verify_move_template(self.default_build_dir, stack_path)
+
+    def _verify_build_artifact(self, build_dir, function_full_path):
+        self.assertTrue(build_dir.exists(), "Build directory should be created")
+
+        build_dir_files = os.listdir(str(build_dir))
+        self.assertIn("template.yaml", build_dir_files)
+        # full_path is always posix path
+        path_components = posixpath.split(function_full_path)
+        artifact_path = Path(build_dir, *path_components)
+        self.assertTrue(artifact_path.exists())
+
+    def _verify_move_template(self, build_dir, stack_path):
+        path_components = posixpath.split(stack_path)
+        stack_build_dir_path = Path(build_dir, Path(*path_components), "template.yaml")
+        self.assertTrue(stack_build_dir_path.exists())
+
+    def _verify_process_code_and_output(self, command_result, function_full_paths, layer_full_path):
+        self.assertEqual(command_result.process.returncode, 0)
+        # check HelloWorld and HelloMars functions are built in the same build
+        for function_full_path in function_full_paths:
+            self.assertRegex(
+                command_result.stderr.decode("utf-8"),
+                f"Building codeuri: .* runtime: .* metadata: .* functions: \\[.*'{function_full_path}'.*\\]",
+            )
+        self.assertIn(
+            f"Building layer '{layer_full_path}'",
+            command_result.stderr.decode("utf-8"),
+        )
+
+    def _verify_invoke_built_functions(self, template_path, functions, error_message):
+        """
+        Invoke the function, if error_message is not None, the invoke should fail.
+        """
+        for function_logical_id in functions:
+            LOG.info("Invoking built function '{}'".format(function_logical_id))
+
+            cmdlist = [
+                self.cmd,
+                "local",
+                "invoke",
+                function_logical_id,
+                "-t",
+                str(template_path),
+                "--no-event",
+            ]
+
+            process_execute = run_command(cmdlist)
+            process_execute.process.wait()
+
+            process_stderr = process_execute.stderr.decode("utf-8")
+            if error_message:
+                self.assertNotEqual(0, process_execute.process.returncode)
+                self.assertIn(error_message, process_stderr)
+            else:
+                self.assertEqual(0, process_execute.process.returncode)
diff --git a/tests/integration/buildcmd/test_build_cmd.py b/tests/integration/buildcmd/test_build_cmd.py
--- a/tests/integration/buildcmd/test_build_cmd.py
+++ b/tests/integration/buildcmd/test_build_cmd.py
@@ -17,6 +17,7 @@
     CachedBuildIntegBase,
     BuildIntegRubyBase,
     NestedBuildIntegBase,
+    IntrinsicIntegBase,
 )
 from tests.testing_utils import (
     IS_WINDOWS,
@@ -1812,3 +1813,59 @@ def test_nested_build(self, use_container, cached, parallel):
                     ("LocalNestedStack/Function2", {"pi": "3.14"}),
                 ],
             )
+
+
+@parameterized_class(
+    ("template", "stack_paths", "layer_full_path", "function_full_paths", "invoke_error_message"),
+    [
+        (
+            os.path.join("nested-with-intrinsic-functions", "template-pass-down.yaml"),
+            ["", "AppUsingRef", "AppUsingJoin"],
+            "MyLayerVersion",
+            ["AppUsingRef/FunctionInChild", "AppUsingJoin/FunctionInChild"],
+            # Note(xinhol), intrinsic function passed by parameter are resolved as string,
+            # therefore it is being treated as an Arn, it is a bug in intrinsic resolver
+            "Invalid Layer Arn",
+        ),
+        (
+            os.path.join("nested-with-intrinsic-functions", "template-pass-up.yaml"),
+            ["", "ChildApp"],
+            "ChildApp/MyLayerVersion",
+            ["FunctionInRoot"],
+            # for this pass-up use case, since we are not sure whether there are valid local invoke cases out there,
+            # so we don't want to block customers from local invoking it.
+            None,
+        ),
+    ],
+)
+class TestBuildPassingLayerAcrossStacks(IntrinsicIntegBase):
+    @pytest.mark.flaky(reruns=3)
+    def test_nested_build(self):
+        if SKIP_DOCKER_TESTS:
+            self.skipTest(SKIP_DOCKER_MESSAGE)
+
+        """
+        Build template above and verify that each function call returns as expected
+        """
+        cmdlist = self.get_command_list(
+            use_container=True,
+            cached=True,
+            parallel=True,
+        )
+
+        LOG.info("Running Command: %s", cmdlist)
+        LOG.info(self.working_dir)
+
+        command_result = run_command(cmdlist, cwd=self.working_dir)
+
+        if not SKIP_DOCKER_TESTS:
+            self._verify_build(
+                self.function_full_paths,
+                self.layer_full_path,
+                self.stack_paths,
+                command_result,
+            )
+
+            self._verify_invoke_built_functions(
+                self.built_template, self.function_full_paths, self.invoke_error_message
+            )
diff --git a/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/child-pass-down.yaml b/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/child-pass-down.yaml
new file mode 100644
--- /dev/null
+++ b/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/child-pass-down.yaml
@@ -0,0 +1,16 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Transform: AWS::Serverless-2016-10-31
+
+Parameters:
+  Layer:
+    Type: String
+
+Resources:
+  FunctionInChild:
+    Type: AWS::Serverless::Function
+    Properties:
+      CodeUri: ../Python
+      Handler: main.handler
+      Runtime: python3.8
+      Layers:
+        - !Ref Layer
diff --git a/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/child-pass-up.yaml b/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/child-pass-up.yaml
new file mode 100644
--- /dev/null
+++ b/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/child-pass-up.yaml
@@ -0,0 +1,18 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Transform: AWS::Serverless-2016-10-31
+
+Resources:
+  MyLayerVersion:
+    Type: AWS::Serverless::LayerVersion
+    Properties:
+      LayerName: MyLayerInChild
+      ContentUri: ../PyLayer
+      CompatibleRuntimes:
+        - python3.8
+    Metadata:
+      BuildMethod: python3.8
+
+Outputs:
+  LayerVersion:
+    Value: !Ref MyLayerVersion
+    Description: MyLayerVersion in child.yaml
diff --git a/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/template-pass-down.yaml b/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/template-pass-down.yaml
new file mode 100644
--- /dev/null
+++ b/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/template-pass-down.yaml
@@ -0,0 +1,27 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Transform: AWS::Serverless-2016-10-31
+
+Resources:
+  MyLayerVersion:
+    Type: AWS::Serverless::LayerVersion
+    Properties:
+      LayerName: MyLayerInRoot
+      ContentUri: ../PyLayer
+      CompatibleRuntimes:
+        - python3.8
+    Metadata:
+      BuildMethod: python3.8
+
+  AppUsingRef:
+    Type: AWS::Serverless::Application
+    Properties:
+      Location: ./child-pass-down.yaml
+      Parameters:
+        Layer: !Ref MyLayerVersion
+
+  AppUsingJoin:
+    Type: AWS::Serverless::Application
+    Properties:
+      Location: ./child-pass-down.yaml
+      Parameters:
+        Layer: !Ref MyLayerVersion
\ No newline at end of file
diff --git a/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/template-pass-up.yaml b/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/template-pass-up.yaml
new file mode 100644
--- /dev/null
+++ b/tests/integration/testdata/buildcmd/nested-with-intrinsic-functions/template-pass-up.yaml
@@ -0,0 +1,17 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Transform: AWS::Serverless-2016-10-31
+
+Resources:
+  ChildApp:
+    Type: AWS::Serverless::Application
+    Properties:
+      Location: ./child-pass-up.yaml
+
+  FunctionInRoot:
+    Type: AWS::Serverless::Function
+    Properties:
+      CodeUri: ../Python
+      Handler: main.handler
+      Runtime: python3.8
+      Layers:
+        - !GetAtt ChildApp.Outputs.LayerVersion
\ No newline at end of file
diff --git a/tests/unit/commands/local/lib/test_provider.py b/tests/unit/commands/local/lib/test_provider.py
--- a/tests/unit/commands/local/lib/test_provider.py
+++ b/tests/unit/commands/local/lib/test_provider.py
@@ -49,8 +49,9 @@ class TestLayerVersion(TestCase):
         ]
     )
     def test_invalid_arn(self, arn):
+        layer = LayerVersion(arn, None)  # creation of layer does not raise exception
         with self.assertRaises(InvalidLayerVersionArn):
-            LayerVersion(arn, None)
+            layer.version, layer.name
 
     def test_layer_version_returned(self):
         layer_version = LayerVersion("arn:aws:lambda:region:account-id:layer:layer-name:1", None)