Repository: localstack/localstack

Issue:
bug: pod save fails when large lambdas are uploaded. (2.1.1.dev)
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Localstack prints a python stack error when attempting to save a large lambda (>1G) via "pod save".   It references S3, but that is a red-herring.  Once this error is encountered, the pod save exits leaving other services absent from the file.

### Expected Behavior

A services should be saved to the file.

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

localstack start -d
awslocal lambda create-function  --function-name kinesisToElasticSearch  --zip-file fileb://kinesisToElasticSearch.zip --handler index.handler --runtime nodejs12.x --role arn:aws:iam::000000000000:role/lambda-role --region us-east-1
localstack pod save file://$PWD/localstack.pod


### Environment

```markdown
- OS:MacOS Ventura 13.2.1
- LocalStack: 2.0.1.dev
```


### Anything else?

Note if you delete the lambda, the problem still occurs.   I have not determined the exact limit on the lambda zip file.

Output

2023-04-04T23:51:16.437  WARN --- [-functhread3] hypercorn.error            : ASGI Framework Lifespan error, continuing without Lifespan support
2023-04-04T23:51:16.437  WARN --- [-functhread3] hypercorn.error            : ASGI Framework Lifespan error, continuing without Lifespan support
2023-04-04T23:51:16.438  INFO --- [-functhread3] hypercorn.error            : Running on https://0.0.0.0:4566 (CTRL + C to quit)
2023-04-04T23:51:16.438  INFO --- [-functhread3] hypercorn.error            : Running on https://0.0.0.0:4566 (CTRL + C to quit)
Ready.
2023-04-04T23:52:32.051  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS lambda.CreateFunction => 201
2023-04-04T23:52:32.126  INFO --- [   asgi_gw_2] localstack.request.aws     : AWS sts.AssumeRole => 200
2023-04-04T23:52:43.746  INFO --- [   asgi_gw_2] localstack.request.http    : GET / => 200
2023-04-04T23:52:43.797 ERROR --- [   asgi_gw_0] l.pods.manager             : Error while saving state of service s3 into pod
Traceback (most recent call last):
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_persistence/pods/manager.py", line 15, in extract_into
try:A.accept_state_visitor(E)
File "/opt/code/localstack/localstack/services/plugins.py", line 145, in accept_state_visitor
ReflectionStateLocator(service=self.name()).accept_state_visitor(visitor)
File "/opt/code/localstack/localstack/state/inspect.py", line 133, in accept_state_visitor
visitor.visit(attribute)
File "/usr/local/lib/python3.10/functools.py", line 926, in _method
return method.__get__(obj, cls)(*args, **kwargs)
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_persistence/pods/save.py", line 27, in _
G=os.path.join(constants.API_STATES_DIRECTORY,C,B,E,constants.MOTO_BACKEND_STATE_FILE)
File "/usr/local/lib/python3.10/posixpath.py", line 90, in join
genericpath._check_arg_types('join', a, *p)
File "/usr/local/lib/python3.10/genericpath.py", line 152, in _check_arg_types
raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'
2023-04-04T23:52:43.805  INFO --- [   asgi_gw_0] localstack.request.http    : GET /_localstack/pods/state => 200
2023-04-04T23:52:44.031  INFO --- [   asgi_gw_2] localstack.request.http    : GET /_localstack/pods/environment => 200


Discussion:
The example shows a Node lambda, but it doesn't matter.   Same problem happens regardless of the lambda type.     Note also this is a simplification of a much more complicated deployment, which is how I know the pod save file is missing services.
This error only happens when the jar/zip file is fairly large (> 1G).   Smaller files seem to work fine.
FWIW: Repo'd on 2.1.1.dev.   This appears to be caused by a 16M lambda layer.

> localstack_main  | 2023-05-26T14:11:02.659  INFO --- [   asgi_gw_2] localstack.request.http    : GET / => 200
> localstack_main  | 2023-05-26T14:11:02.742 ERROR --- [   asgi_gw_1] l.pods.manager             : Error while saving state of service s3 into pod
> localstack_main  | Traceback (most recent call last):
> localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_persistence/pods/manager.py", line 15, in extract_into
> localstack_main  |     try:A.accept_state_visitor(E)
> localstack_main  |   File "/opt/code/localstack/localstack/services/plugins.py", line 145, in accept_state_visitor
> localstack_main  |     ReflectionStateLocator(service=self.name()).accept_state_visitor(visitor)
> localstack_main  |   File "/opt/code/localstack/localstack/state/inspect.py", line 133, in accept_state_visitor
> localstack_main  |     visitor.visit(attribute)
> localstack_main  |   File "/usr/local/lib/python3.10/functools.py", line 926, in _method
> localstack_main  |     return method.__get__(obj, cls)(*args, **kwargs)
> localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_persistence/pods/save.py", line 27, in _
> localstack_main  |     G=os.path.join(constants.API_STATES_DIRECTORY,C,B,E,constants.MOTO_BACKEND_STATE_FILE)
> localstack_main  |   File "/usr/local/lib/python3.10/posixpath.py", line 90, in join
> localstack_main  |     genericpath._check_arg_types('join', a, *p)
> localstack_main  |   File "/usr/local/lib/python3.10/genericpath.py", line 152, in _check_arg_types
> localstack_main  |     raise TypeError(f'{funcname}() argument must be str, bytes, or '
> localstack_main  | TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'
> localstack_main  | 2023-05-26T14:11:02.767  INFO --- [   asgi_gw_1] localstack.request.http    : GET /_localstack/pods/state => 200
> localstack_main  | 2023-05-26T14:11:03.097  INFO --- [   asgi_gw_5] localstack.request.http    : GET /_localstack/pods/environment => 200
For kicks I tried to delete the layer, but that exposed another problem:

> localstack_main  | 2023-05-26T14:21:49.145  INFO --- [   asgi_gw_3] localstack.request.aws     : AWS lambda.ListLayers => 200
> localstack_main  | 2023-05-26T14:23:10.306  INFO --- [   asgi_gw_4] localstack.request.aws     : AWS lambda.DeleteLayerVersion => 204
> localstack_main  | 2023-05-26T14:23:14.270 ERROR --- [   asgi_gw_1] l.aws.handlers.logging     : exception during call chain: list index out of range
> localstack_main  | 2023-05-26T14:23:14.278  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS lambda.ListLayers => 500 (InternalError)
> localstack_main  | 2023-05-26T14:23:15.214 ERROR --- [   asgi_gw_1] l.aws.handlers.logging     : exception during call chain: list index out of range
> localstack_main  | 2023-05-26T14:23:15.216  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS lambda.ListLayers => 500 (InternalError)
> localstack_main  | 2023-05-26T14:23:16.948 ERROR --- [   asgi_gw_2] l.aws.handlers.logging     : exception during call chain: list index out of range
> localstack_main  | 2023-05-26T14:23:16.949  INFO --- [   asgi_gw_2] localstack.request.aws     : AWS lambda.ListLayers => 500 (InternalError)
Any update on this issue? I am facing the same problem
I am seeing this but with a 13.5 MiB file used in a lambda.
Same. My file is too small. 
For now my S3 contents saved in pod file on running command but it throws error. 
On next load, S3 contents are retained other then the lambda function.
I'm also seeing this on the pro version with persistence enabled where the lambda function isn't saved.
So in save.py at line 27, value of C is None that's why it is breaking. I just troubleshoot it by hardcoding a value 000000000000 for C just to test and it works fine. Now waiting for fix from team
G=os.path.join(constants.API_STATES_DIRECTORY,C,B,E,constants.MOTO_BACKEND_STATE_FILE)

So for now the solution is not to upload any file on S3 (either by CLI or by code) as multipart. It will throw this error. Use PutObject to upload any file to S3 and it will not break 
Harder to control when using terraform.

> So for now the solution is not to upload any file on S3 (either by CLI or by code) as multipart. It will throw this error. Use PutObject to upload any file to S3 and it will not break



Resolving patch:
diff --git a/localstack/services/s3/provider.py b/localstack/services/s3/provider.py
--- a/localstack/services/s3/provider.py
+++ b/localstack/services/s3/provider.py
@@ -788,14 +788,32 @@ def upload_part(self, context: RequestContext, request: UploadPartRequest) -> Up
                 "The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.",
                 UploadId=upload_id,
             )
-        elif (part_number := request.get("PartNumber", 0)) < 1:
+        elif (part_number := request.get("PartNumber", 0)) < 1 or part_number >= 10000:
             raise InvalidArgument(
                 "Part number must be an integer between 1 and 10000, inclusive",
                 ArgumentName="partNumber",
                 ArgumentValue=part_number,
             )
 
-        response: UploadPartOutput = call_moto(context)
+        part = body.read() if (body := request.get("Body")) else b""
+
+        # we are directly using moto backend and not calling moto because to get the response, moto calls
+        # key.response_dict, which in turns tries to access the tags of part, indirectly creating a BackendDict
+        # with an account_id set to None (because moto does not set an account_id to the FakeKey representing a Part)
+        key = moto_backend.upload_part(bucket_name, upload_id, part_number, part)
+        response = UploadPartOutput(ETag=key.etag)
+
+        if key.checksum_algorithm is not None:
+            response[f"Checksum{key.checksum_algorithm.upper()}"] = key.checksum_value
+
+        if key.encryption is not None:
+            response["ServerSideEncryption"] = key.encryption
+            if key.encryption == "aws:kms" and key.kms_key_id is not None:
+                response["SSEKMSKeyId"] = key.encryption
+
+        if key.encryption == "aws:kms" and key.bucket_key_enabled is not None:
+            response["BucketKeyEnabled"] = key.bucket_key_enabled
+
         return response
 
     @handler("ListMultipartUploads", expand=False)


Test patch: