Repository: localstack/localstack

Issue:
bug: Lambda improperly parses docker images with digest
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Using the serverless framework, when deploying a lambda function that uses a custom container runtime, the image is pushed to localstack with an "ImageUri" value that includes the docker image's digest (e.g. `localhost:4513/serverless-test-local@sha256:306d92[TRUNCATED]`). When executing that lambda function, fullstack uses the docker url that includes the digest, and improperly treats the portion after `:` as an image tag (https://github.com/localstack/localstack/blob/master/localstack/utils/docker_utils.py#L1203), causing the function to fail.

### Expected Behavior

Localstack should pull the docker image with the properly formatted digest and successfully execute the lambda function.

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

    localstack start

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
Sample serverless.yml
```
service: test
plugins:
  - serverless-localstack

frameworkVersion: '2'

provider:
  name: aws
  lambdaHashingVersion: 20201221
  ecr:
    images:
      test:
        path: ./test

custom:
  localstack:
    debug: true
    stages:
      - local
    host: http://localhost
    edgePort: 4566  
    autostart: true 
    docker:
      sudo: False

functions:
  test:
    image:
      name: test
```

Execute function
```
sls deploy --stage local
awslocal lambda invoke --function-name test-local-test /dev/null
```

Localstack error
```
2021-11-08T19:25:08:INFO:localstack.services.awslambda.lambda_executors: Running lambda: arn:aws:lambda:us-east-1:000000000000:function:test-local-test
2021-11-08T19:25:08:DEBUG:localstack_ext.services.awslambda.lambda_extended: Putting invocation event (request ID d01c1e28) for Lambda 'arn:aws:lambda:us-east-1:000000000000:function:test-local-test' to queue
2021-11-08T19:25:08:DEBUG:localstack_ext.services.awslambda.lambda_launcher: Executing custom image execution for image localhost:4513/serverless-test-local@sha256:306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221
2021-11-08T19:25:11:DEBUG:localstack.utils.docker_utils: Pulling image: localhost:4513/serverless-test-local@sha256:306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221
2021-11-08T19:25:11:DEBUG:localstack.utils.docker_utils: Repository: localhost:4513/serverless-test-local@sha256 Tag: 306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221
2021-11-08T19:25:12:ERROR:localstack_ext.services.awslambda.lambda_extended: Unable to invoke Lambda "arn:aws:lambda:us-east-1:000000000000:function:test-local-test": 
Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/create?tag=306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221&fromImage=localhost%3A4513%2Fserverless-test-local

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/localstack/utils/docker_utils.py", line 1208, in pull_image
    self.client().images.pull(repository, tag)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/models/images.py", line 447, in pull
    repository, tag=tag, stream=True, all_tags=all_tags, **kwargs
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/image.py", line 430, in pull
    self._raise_for_status(response)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221&fromImage=localhost%3A4513%2Fserverless-test-local: Not Found ("manifest for localhost:4513/serverless-test-local:306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221 not found: manifest unknown: manifest unknown")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_extended.py", line 448, in do_run_lambda_executor
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_launcher.py", line 222, in docker_separate_execute_in_container
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_launcher.py", line 189, in custom_image_execute_in_container
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_launcher.py", line 148, in get_image_entrypoint_cmd
  File "/opt/code/localstack/localstack/utils/docker_utils.py", line 1212, in pull_image
    raise ContainerException()
localstack.utils.docker_utils.ContainerException
2021-11-08T19:25:12:INFO:localstack_ext.services.awslambda.lambda_extended: Lambda container output: None
None
2021-11-08T19:25:12:INFO:localstack.services.awslambda.lambda_api: Error executing Lambda function arn:aws:lambda:us-east-1:000000000000:function:test-local-test:  Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/create?tag=306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221&fromImage=localhost%3A4513%2Fserverless-test-local

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/localstack/utils/docker_utils.py", line 1208, in pull_image
    self.client().images.pull(repository, tag)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/models/images.py", line 447, in pull
    repository, tag=tag, stream=True, all_tags=all_tags, **kwargs
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/image.py", line 430, in pull
    self._raise_for_status(response)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221&fromImage=localhost%3A4513%2Fserverless-test-local: Not Found ("manifest for localhost:4513/serverless-test-local:306d925cd4f0188bcb9808c7715fc444dff3eee356415e58651db25304825221 not found: manifest unknown: manifest unknown")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 813, in run_lambda
    lock_discriminator=lock_discriminator,
  File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 429, in execute
    return do_execute()
  File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 419, in do_execute
    return _run(func_arn=func_arn)
  File "/opt/code/localstack/localstack/utils/cloudwatch/cloudwatch_util.py", line 158, in wrapped
    raise e
  File "/opt/code/localstack/localstack/utils/cloudwatch/cloudwatch_util.py", line 154, in wrapped
    result = func(*args, **kwargs)
  File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 406, in _run
    raise e
  File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 402, in _run
    result = self._execute(lambda_function, inv_context)
  File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 710, in _execute
    result = self.run_lambda_executor(lambda_function=lambda_function, inv_context=inv_context)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_extended.py", line 538, in run_lambda_executor
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_extended.py", line 436, in run_lambda_executor
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_extended.py", line 448, in do_run_lambda_executor
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_launcher.py", line 222, in docker_separate_execute_in_container
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_launcher.py", line 189, in custom_image_execute_in_container
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/localstack_ext/services/awslambda/lambda_launcher.py", line 148, in get_image_entrypoint_cmd
  File "/opt/code/localstack/localstack/utils/docker_utils.py", line 1212, in pull_image
    raise ContainerException()
localstack.utils.docker_utils.ContainerException
```


### Environment

```markdown
- OS: MacOS Big Sur 11.1
- LocalStack: 0.12.20
```


### Anything else?

_No response_


Discussion:
Hi @JacobReynolds !
As a quick workaround, you could use `LEGACY_DOCKER_CLIENT=1`, to circumvent the problem.
I will notify you when a proper fix is in place.
Hey @dfangl thanks for the response! It looks like `LEGACY_DOCKER_CLIENT` isn't tracked [here](https://github.com/localstack/localstack/blob/d46901d4aec96ff7cb90b05c16b23bd74f838c3c/localstack/config.py#L312), so the CLI doesn't pass it to the docker container.  I've added a pull request for that #4882 

Resolving patch:
diff --git a/localstack/utils/docker_utils.py b/localstack/utils/docker_utils.py
--- a/localstack/utils/docker_utils.py
+++ b/localstack/utils/docker_utils.py
@@ -1199,13 +1199,9 @@ def copy_from_container(
     def pull_image(self, docker_image: str) -> None:
         LOG.debug("Pulling image: %s", docker_image)
         # some path in the docker image string indicates a custom repository
-        path_split = docker_image.rpartition("/")
-        image_split = path_split[2].partition(":")
-        repository = f"{path_split[0]}{path_split[1]}{image_split[0]}"
-        tag = image_split[2]
         try:
-            LOG.debug("Repository: %s Tag: %s", repository, tag)
-            self.client().images.pull(repository, tag)
+            LOG.debug("Repository: %s", docker_image)
+            self.client().images.pull(docker_image)
         except ImageNotFound:
             raise NoSuchImage(docker_image)
         except APIError:


Test patch:
diff --git a/tests/integration/docker_utils/test_docker.py b/tests/integration/docker_utils/test_docker.py
--- a/tests/integration/docker_utils/test_docker.py
+++ b/tests/integration/docker_utils/test_docker.py
@@ -597,6 +597,42 @@ def test_pull_non_existent_docker_image(self, docker_client: ContainerClient):
         with pytest.raises(NoSuchImage):
             docker_client.pull_image("localstack_non_existing_image_for_tests")
 
+    def test_pull_docker_image_with_tag(self, docker_client: ContainerClient):
+        try:
+            docker_client.get_image_cmd("alpine")
+            safe_run([config.DOCKER_CMD, "rmi", "alpine"])
+        except ContainerException:
+            pass
+        with pytest.raises(NoSuchImage):
+            docker_client.get_image_cmd("alpine")
+        docker_client.pull_image("alpine:3.13")
+        assert "/bin/sh" == docker_client.get_image_cmd("alpine:3.13").strip()
+        assert "alpine:3.13" in docker_client.inspect_image("alpine:3.13")["RepoTags"]
+
+    def test_pull_docker_image_with_hash(self, docker_client: ContainerClient):
+        try:
+            docker_client.get_image_cmd("alpine")
+            safe_run([config.DOCKER_CMD, "rmi", "alpine"])
+        except ContainerException:
+            pass
+        with pytest.raises(NoSuchImage):
+            docker_client.get_image_cmd("alpine")
+        docker_client.pull_image(
+            "alpine@sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a"
+        )
+        assert (
+            "/bin/sh"
+            == docker_client.get_image_cmd(
+                "alpine@sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a"
+            ).strip()
+        )
+        assert (
+            "alpine@sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a"
+            in docker_client.inspect_image(
+                "alpine@sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a"
+            )["RepoDigests"]
+        )
+
     def test_run_container_automatic_pull(self, docker_client: ContainerClient):
         try:
             safe_run([config.DOCKER_CMD, "rmi", "alpine"])