Repository: aws/aws-sam-cli

Issue:
`sam build` broken after upgrading SAM CLI when using ECR (backward compatibility issue)
### Description:
I have a SAM application with an AWS ECR repository configured as the storage for the app snapshots/releases.

So far I was using just `sam build` and `sam deploy` to deploy my SAM application. It used to work with SAM CLI 1.24.1.

Unfortunately after I upgraded to SAM CLI 1.27.2 it stopped working. Now `sam build` says:

> The resource AWS::Serverless::Function 'HelloFunction' has specified ECR registry image for ImageUri. It will not be built and SAM CLI does not support invoking it locally.

Upgrading SAM CLI to the current latest version 1.30.0 doesn't help either.

**This is a breaking change.** Not sure if it was announced anywhere. Also it seems the problem was introduced as part of #2934.

Honestly I don't know what I should do now and how I should fix it. For now I can stick to the SAM CLI version that works for me but this is only a temporary solution.

### Steps to reproduce:
Create a SAM app with a template including:
```
Resources:
  HelloFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageUri: <account-number>.dkr.ecr.<region>.amazonaws.com/hello-app
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./app
      DockerTag: python3.8-v1
```

Use `sam build` and then guided deploy with SAM CLI 1.24.1. Then upgrade to SAM CLI 1.27.2 - `sam build` will stop working.


### Observed result:
`sam build` doesn't build the image anymore and `sam deploy` doesn't push the image to ECR. As a result `sam deploy` fails.



### Expected result:
`sam build` should build the image, as it used to. `sam deploy` should push the image to ECR and update the app without issues.



### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Ubuntu 20.04
2. `sam --version`: 1.24.1, 1.27.2, 1.30.0
3. AWS region: Ireland (eu-west-1)



Discussion:
Tagging @alexisfacques 
Heya

Given the fact, the `ImageUri` parameter is irrelevant in the build/deploy process (the final ECR Repo / Image URI to which you publish the image must be specified via deploy config parameters), it was agreed off-side in #2934 to skip the image build when the `ImageUri` parameter is a valid ECR image.
This allows you to deploy pre-packaged `PackageType: Image` functions with SAM, the same way you may deploy `PackageType: Zip` images pointing to a valid S3 location.

Changing the `ImageUri` parameter to something else than the ECR repo image name will fix the problem (e.g. `hello-app:latest`). This will be the tag used by SAM to build Docker image in your local environment, prior to deploying it
Hi,

Thanks for providing some background. Next time I'd suggest making such new behavior optional so that existing users could adapt to it over time. Or you could just bump the major version of the tool to indicate a breaking change and document it properly in the release notes.

Anyway I was just looking at the description of [ImageUri](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-imageuri) parameter:
> The URI of the Amazon Elastic Container Registry (Amazon ECR) repository for the Lambda function's container image. This property only applies if the PackageType property is set to Image, otherwise it is ignored.

If setting `ImageUri` to something other than an ECR repo URL is supported then it should be mentioned here.

Moreover, while I understand that the change you introduced allows someone to use a pre-packaged and pre-published Docker image, the docs should still make it clear on how to use `sam build`/`sam deploy` together with ECR to achieve the simpler use-case where all you want to do is to use SAM CLI to do the build/push steps for you.

I checked other parts of the documentation related to building images with SAM apps and I must admit I haven't found a clear description of how this should be done. For example see [Building applications, Example 2: Container image](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-build.html#examples-container-image-1) - it shows you an exemplary SAM template and a Dockerfile, but it doesn't say how it should (could) be built and published.
Can you explain more about why `sam deploy` is failing? What message do you get on deployment?
The error message is:

```
Resource handler returned message: "Source image <account-nr>.dkr.ecr.eu-west-1.amazonaws.com/hello-world is not valid. Provide a valid source image.  (Service: Lambda, Status Code: 400, Request ID: <request-id>, Extended Request ID: null)" (RequestToken: <request-token>, HandlerErrorCode: InvalidRequest)
```

Below is a full log:

```
$ sam deploy

	Deploying with following values
	===============================
	Stack name                   : hello-world
	Region                       : eu-west-1
	Confirm changeset            : True
	Deployment image repository  : 
                                       {
                                           "HelloFunction": "<account-number>.dkr.ecr.eu-west-1.amazonaws.com/hello-world"
                                       }
	Deployment s3 bucket         : <bucket-name>
	Capabilities                 : ["CAPABILITY_IAM"]
	Parameter overrides          : {}
	Signing Profiles             : {}

Initiating deployment
=====================
File with same data already exists at hello-world/01dab9ed7445d678ea5d297a4b4e801c.template, skipping upload

Waiting for changeset to be created..

CloudFormation stack changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                  LogicalResourceId                                          ResourceType                                               Replacement                                              
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
* Modify                                                   HelloFunctionHelloPermission                               AWS::Lambda::Permission                                    Conditional                                              
* Modify                                                   HelloFunction                                              AWS::Lambda::Function                                      False                                                    
* Modify                                                   ServerlessHttpApiApiGatewayDefaultStage                    AWS::ApiGatewayV2::Stage                                   Conditional                                              
* Modify                                                   ServerlessHttpApi                                          AWS::ApiGatewayV2::Api                                     Conditional                                              
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:eu-west-1:<account-nr>:changeSet/samcli-deploy1630913936/1e0f72bc-453d-44f4-9b6f-5f34946eb8fb


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2021-09-06 09:39:09 - Waiting for stack create/update to complete

CloudFormation events from changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                             ResourceType                                               LogicalResourceId                                          ResourceStatusReason                                     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE_IN_PROGRESS                                         AWS::Lambda::Function                                      HelloFunction                                        -                                                        
UPDATE_ROLLBACK_IN_PROGRESS                                AWS::CloudFormation::Stack                                 hello-stack                                          The following resource(s) failed to update:              
                                                                                                                                                                                 [HelloFunction].                                   
UPDATE_FAILED                                              AWS::Lambda::Function                                      HelloFunction                                        Resource handler returned message: "Source image         
                                                                                                                                                                                 <account-nr>.dkr.ecr.eu-west-1.amazonaws.com/hello-world is not valid. Provide a valid source image.  
                                                                                                                                                                                 (Service: Lambda, Status Code: 400, Request ID:          
                                                                                                                                                                                 b31c6317-5934-4cd2-9c0d-461a0883344a, Extended Request   
                                                                                                                                                                                 ID: null)" (RequestToken:                                
                                                                                                                                                                                 dbae00b4-952f-0818-5a44-6cb126006e97, HandlerErrorCode:  
                                                                                                                                                                                 InvalidRequest)                                          
UPDATE_IN_PROGRESS                                         AWS::Lambda::Function                                      HelloFunction                                        -                                                       
UPDATE_COMPLETE                                            AWS::Lambda::Function                                      HelloFunction                                        -                                                        
UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS               AWS::CloudFormation::Stack                                 hello-stack                                          -                                                        
UPDATE_ROLLBACK_COMPLETE                                   AWS::CloudFormation::Stack                                 hello-stack                                          -                                                        
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Error: Failed to create/update the stack: hello-stack, Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "UPDATE_ROLLBACK_COMPLETE" at least once
```

Tested with `sam deploy` using SAM CLI 1.27.2, of course using the original SAM template, with a full ECR repository URI in `ImageUri`.
Are you able to reproduce the issue?
@romanek-adam Sorry about the confusion.

When you provide the Metadata below and set the package type to `Image`, the `ImageUri` is ignored during the `sam build` process in previous version of sam-cli. The built artifact (`.aws-sam/build/template.yaml`) will have `ImageUri` with values like "ImageUri: helloworldfunction:nodejs14.x-v1" and `sam package` or `sam deploy` will upload the built imagine and replace it with uploaded image uri. **So the previously provided ImageUri has no effect**

```
Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./app
      DockerTag: python3.8-v1
```

Related doc: 

> Note: If the PackageType property is set to Image, then either ImageUri is required, or you must build your application with necessary Metadata entries in the AWS SAM template file.  https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-imageuri

After the change in #2934, we skipped building functions with a "valid" `ImageUri`, and this change broke your workflow, sorry about the inconvenience. 

You can remove `ImageUri` since the value is not used. Then do `sam build` then `sam deploy` (or `sam package`) to test whether it fixes your issue or not.

We will work on some documentation change to avoid such confusion in the future.

Resolving patch:
diff --git a/samcli/lib/providers/sam_function_provider.py b/samcli/lib/providers/sam_function_provider.py
--- a/samcli/lib/providers/sam_function_provider.py
+++ b/samcli/lib/providers/sam_function_provider.py
@@ -144,8 +144,12 @@ def _extract_functions(
                             SamFunctionProvider._warn_code_extraction(resource_type, name, code_property_key)
                         continue
 
-                    if resource_package_type == IMAGE and SamBaseProvider._is_ecr_uri(
-                        resource_properties.get(image_property_key)
+                    if (
+                        resource_package_type == IMAGE
+                        and SamBaseProvider._is_ecr_uri(resource_properties.get(image_property_key))
+                        and not SamFunctionProvider._metadata_has_necessary_entries_for_image_function_to_be_built(
+                            resource_metadata
+                        )
                     ):
                         # ImageUri can be an ECR uri, which is not supported
                         if not ignore_code_extraction_warnings:
@@ -460,3 +464,19 @@ def get_resources_by_stack_path(self, stack_path: str) -> Dict:
         if not candidates:
             raise RuntimeError(f"Cannot find resources with stack_path = {stack_path}")
         return candidates[0]
+
+    @staticmethod
+    def _metadata_has_necessary_entries_for_image_function_to_be_built(metadata: Optional[Dict[str, Any]]) -> bool:
+        """
+        > Note: If the PackageType property is set to Image, then either ImageUri is required,
+          or you must build your application with necessary Metadata entries in the AWS SAM template file.
+          https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-imageuri
+
+        When ImageUri and Metadata are both provided, we will try to determine whether to treat the function
+        as to be built or to be skipped. When we skip it whenever "ImageUri" is provided,
+        we introduced a breaking change https://github.com/aws/aws-sam-cli/issues/3239
+
+        This function is used to check whether there are the customers have "intention" to
+        let AWS SAM CLI to build this image function.
+        """
+        return isinstance(metadata, dict) and bool(metadata.get("DockerContext"))


Test patch:
diff --git a/tests/unit/commands/local/lib/test_sam_function_provider.py b/tests/unit/commands/local/lib/test_sam_function_provider.py
--- a/tests/unit/commands/local/lib/test_sam_function_provider.py
+++ b/tests/unit/commands/local/lib/test_sam_function_provider.py
@@ -95,6 +95,15 @@ class TestSamFunctionProviderEndToEnd(TestCase):
                     "PackageType": IMAGE,
                 },
             },
+            "SamFuncWithImage4": {
+                # ImageUri is unsupported ECR location, but metadata is still provided, build
+                "Type": "AWS::Serverless::Function",
+                "Properties": {
+                    "ImageUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/myrepo:myimage",
+                    "PackageType": IMAGE,
+                },
+                "Metadata": {"DockerTag": "tag", "DockerContext": "./image", "Dockerfile": "Dockerfile"},
+            },
             "LambdaFunc1": {
                 "Type": "AWS::Lambda::Function",
                 "Properties": {
@@ -126,6 +135,15 @@ class TestSamFunctionProviderEndToEnd(TestCase):
                     "PackageType": IMAGE,
                 },
             },
+            "LambdaFuncWithImage4": {
+                # ImageUri is unsupported ECR location, but metadata is still provided, build
+                "Type": "AWS::Lambda::Function",
+                "Properties": {
+                    "Code": {"ImageUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/myrepo"},
+                    "PackageType": IMAGE,
+                },
+                "Metadata": {"DockerTag": "tag", "DockerContext": "./image", "Dockerfile": "Dockerfile"},
+            },
             "LambdaFuncWithInlineCode": {
                 "Type": "AWS::Lambda::Function",
                 "Properties": {
@@ -337,6 +355,33 @@ def setUp(self):
                 ),
             ),
             ("SamFuncWithImage3", None),  # imageuri is ecr location, ignored
+            (
+                "SamFuncWithImage4",  # despite imageuri is ecr location, the necessary metadata is still provided, build
+                Function(
+                    name="SamFuncWithImage4",
+                    functionname="SamFuncWithImage4",
+                    runtime=None,
+                    handler=None,
+                    codeuri=".",
+                    memory=None,
+                    timeout=None,
+                    environment=None,
+                    rolearn=None,
+                    layers=[],
+                    events=None,
+                    inlinecode=None,
+                    imageuri="123456789012.dkr.ecr.us-east-1.amazonaws.com/myrepo:myimage",
+                    imageconfig=None,
+                    packagetype=IMAGE,
+                    metadata={
+                        "DockerTag": "tag",
+                        "DockerContext": os.path.join("image"),
+                        "Dockerfile": "Dockerfile",
+                    },
+                    codesign_config_arn=None,
+                    stack_path="",
+                ),
+            ),
             (
                 "SamFuncWithFunctionNameOverride-x",
                 Function(
@@ -416,6 +461,33 @@ def setUp(self):
                 ),
             ),
             ("LambdaFuncWithImage3", None),  # imageuri is a ecr location, ignored
+            (
+                "LambdaFuncWithImage4",  # despite imageuri is ecr location, the necessary metadata is still provided, build
+                Function(
+                    name="LambdaFuncWithImage4",
+                    functionname="LambdaFuncWithImage4",
+                    runtime=None,
+                    handler=None,
+                    codeuri=".",
+                    memory=None,
+                    timeout=None,
+                    environment=None,
+                    rolearn=None,
+                    layers=[],
+                    events=None,
+                    metadata={
+                        "DockerTag": "tag",
+                        "DockerContext": os.path.join("image"),
+                        "Dockerfile": "Dockerfile",
+                    },
+                    inlinecode=None,
+                    imageuri="123456789012.dkr.ecr.us-east-1.amazonaws.com/myrepo",
+                    imageconfig=None,
+                    packagetype=IMAGE,
+                    codesign_config_arn=None,
+                    stack_path="",
+                ),
+            ),
             (
                 "LambdaFuncWithInlineCode",
                 Function(
@@ -595,10 +667,12 @@ def test_get_all_must_return_all_functions(self):
             "SamFunctions",
             "SamFuncWithImage1",
             "SamFuncWithImage2",
+            "SamFuncWithImage4",
             "SamFuncWithInlineCode",
             "SamFuncWithFunctionNameOverride",
             "LambdaFuncWithImage1",
             "LambdaFuncWithImage2",
+            "LambdaFuncWithImage4",
             "LambdaFuncWithInlineCode",
             "LambdaFuncWithLocalPath",
             "LambdaFuncWithFunctionNameOverride",