Repository: localstack/localstack

Issue:
bug: The last version throws the "Expected hash not equal to calculated hash" error for uploading a multi part.
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

I am using the latest tag for the CI/CD process and today it started to fail because unit tests for the S3 provider that cover multipart upload throw the "Expected hash not equal to calculated hash" error on uploading a part. It worked just fine with the old version. The same unit tests work correctly with the real S3 provider as well.

And there is another issue. You fixed the https://github.com/localstack/localstack/issues/8392 bug that I reported previously but there is no version on Docker Hub that has that fix but does not have the new bug I described above. It seems that you rewrite the latest tag in Docker Hub which makes it impossible to rollback to the previous version.

### Expected Behavior

1. The upload part works without throwing an exception.
2. The previous versions of Docker images are available on Docker Hub.

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

1. Initiate multipart upload with the correct MD5 hash.
2. Upload a part for which the hash was calculated.

### Environment

```markdown
- OS: Windows 11
- LocalStack: latest
```


### Anything else?

_No response_


Discussion:
Hi @real-az and sorry for the inconvenience, we just released a bug fix with multipart upload, which apparently triggered this regression not covered by our tests. Could you share a minimal sample to reproduce the issue? Even if a small code snippet, so that we know how to reproduce and fix this issue ASAP. 
I have a PR up that should fix the issue, but I'd like to confirm it would fix your issue for sure. Don't hesitate to send a small replication code snippet. Are you using the .NET SDK? 
I am using .NET SDK. I have reproduced the issue in the small console application. Please find it attached.
[LocalStack.S3.Issue.zip](https://github.com/localstack/localstack/files/12059857/LocalStack.S3.Issue.zip)

@real-az thanks a lot for the reproduction, I can confirm #8704 fixes the issue. Sorry for the overlooking and the regression, our testing client does not support `aws-chunk` requests, which makes it hard to test. I'll try to get this merged as soon as possible. Thanks again for the report! 
Awesome, thank you for the quick response!

Nonetheless, I still think that the behavior that you follow for publishing Docker images is bad. You should follow the semantic versioning as you describe it in your [documentation](https://docs.localstack.cloud/references/docker-images/#image-tags). As you mentioned above, you made a bug fix with multipart upload which should lead to a PATCH version increment in the version number. That did not happen, the latest version was just rewritten. I really do hope that you will start to follow the semantic versioning.
I understand what you mean. It would be hard to create a patch release after every bug fix in our main branch, as we also merged some features that wouldn't fit the semantic versioning of the `patch` release. We're actually releasing a new `2.2` next week. However, I can see we maybe could have created a `2.1.1` release not long after `2.1`.
It is easy to resolve by having the correct flow for branches. For now, you should have the release version 2.1.0 and all backward-compatible bugfixes in your main branch. And yes, every bugfix will cause the new patch version, so you will have versions 2.1.x coming out of you main branch. All the features that you want to have in version 2.2 should reside in the separate release branch 2.2, from which you can produce alpha, beta, rc1, etc. versions if you like. It is better to merge all bug fixes from the main branch to the 2.2 branch.
If you do not follow this or similar approach you cannot say that you are using the semantic versioning to the full, you will follow just a part of it.
On the other hand, if you follow the correct branch flow, you will have the maximum flexibility for new releases and you will provide us the possibility to use whatever version we want to use, especially useful in situations like the one we have right now.
I would have been able to select the previous version in a matter of seconds and have a working CI/CD process in almost no time without a code change. Instead, I had to disable failing unit tests and disable code analysis tools for them. I think I made my point.
Please do not get me wrong, I appreciate you quick response and the fix you made. I just think that the process can be improved and it will benefit both you who create these awesome and super helpful tools, and us who consume them.
Cheers.
The issue is not fixed by the latest version available on Docker Hub. You can still reproduce it using the test project I provided above.

Resolving patch:
diff --git a/localstack/services/s3/provider.py b/localstack/services/s3/provider.py
--- a/localstack/services/s3/provider.py
+++ b/localstack/services/s3/provider.py
@@ -813,7 +813,10 @@ def upload_part(self, context: RequestContext, request: UploadPartRequest) -> Up
 
         body = request.get("Body")
         headers = context.request.headers
-        if body and "aws-chunked" in (headers.get("Content-Encoding") or "").lower():
+        # AWS specifies that the `Content-Encoding` should be `aws-chunked`, but some SDK don't set it.
+        # Rely on the `x-amz-content-sha256` which is a more reliable indicator that the request is streamed
+        content_sha_256 = (headers.get("x-amz-content-sha256") or "").upper()
+        if body and content_sha_256 and content_sha_256.startswith("STREAMING-"):
             # this is a chunked request, we need to properly decode it while setting the key value
             decoded_content_length = int(headers.get("x-amz-decoded-content-length", 0))
             buffer = io.BytesIO()
diff --git a/localstack/services/s3/provider_stream.py b/localstack/services/s3/provider_stream.py
--- a/localstack/services/s3/provider_stream.py
+++ b/localstack/services/s3/provider_stream.py
@@ -125,7 +125,10 @@ def put_object(
         key_object.checksum_algorithm = checksum_algorithm
 
         headers = context.request.headers
-        if "aws-chunked" in (headers.get("Content-Encoding") or "").lower():
+        # AWS specifies that the `Content-Encoding` should be `aws-chunked`, but some SDK don't set it.
+        # Rely on the `x-amz-content-sha256` which is a more reliable indicator that the request is streamed
+        content_sha_256 = (headers.get("x-amz-content-sha256") or "").upper()
+        if content_sha_256 and content_sha_256.startswith("STREAMING-"):
             # this is a chunked request, we need to properly decode it while setting the key value
             decoded_content_length = int(headers.get("x-amz-decoded-content-length", 0))
             key_object.set_value_from_chunked_payload(body, decoded_content_length)
@@ -278,7 +281,10 @@ def upload_part(self, context: RequestContext, request: UploadPartRequest) -> Up
         body = request.get("Body") or BytesIO()
         decoded_content_length = None
         headers = context.request.headers
-        if "aws-chunked" in (headers.get("Content-Encoding") or "").lower():
+        # AWS specifies that the `Content-Encoding` should be `aws-chunked`, but some SDK don't set it.
+        # Rely on the `x-amz-content-sha256` which is a more reliable indicator that the request is streamed
+        content_sha_256 = (headers.get("x-amz-content-sha256") or "").upper()
+        if content_sha_256 and content_sha_256.startswith("STREAMING-"):
             # this is a chunked request, we need to properly decode it while setting the key value
             decoded_content_length = int(headers.get("x-amz-decoded-content-length", 0))
 


Test patch: