Repository: localstack/localstack

Issue:
bug: Additional headers in S3 presigned URL doesn't work
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

TL;DR: When presigned URL contains one of `Content-MD5`, `X-AMZ-Server-Side-Encryption` headers or their combination, it does not work on localstack, but works in AWS. 

The **presigned S3 URL feature** allows to **add additional headers** to the request when using the **Version 4** signature. These headers are then used in the signing mechanism and verified when such URL is used.

**Use case**: When generating an URL to PUT a file into S3 bucket using the `aws-sdk` Node.js API, I add the MD5 hash of the file-to-be-uploaded to ensure the integrity and say that server-side-encryption should be used. Note that these arguments are added through the native options of `aws-sdk` API:

```js
const s3 = new AWS.S3({
  endpoint: 'http://localhost:4566',
  signatureVersion: 'v4', // Required for the presigned URL functionality with extra headers
  s3ForcePathStyle: true, // Do not use custom domains
});

const body = '123456';
const bodyMd5AsBase64 = '4QrcOUm6Wau+VuBX8g+IPg=='

const url = await s3.getSignedUrlPromise('putObject', {
  Bucket: 'test-bucket',
  Key: 'test-file',
  Expires: 3600,
  ServerSideEncryption: 'AES256', // Adds 'X-Amz-Server-Side-Encryption' in query string
  ContentMD5: bodyMD5             // Adds 'Content-MD5' parameter in query string
});

// url: http://localhost:4566/testBucket/testFile
//   ?Content-MD5=4QrcOUm6Wau%2BVuBX8g%2BIPg%3D%3D
//   &X-Amz-Algorithm=AWS4-HMAC-SHA256
//   &X-Amz-Credential=test%2F20220113%2Fus-east-1%2Fs3%2Faws4_request
//   &X-Amz-Date=20220113T142952Z
//   &X-Amz-Expires=3600
//   &X-Amz-Signature=d219a729f06e37d40a136bb5fec777265b1b34e879f9e338d385b39a3760a14f
//   &X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-server-side-encryption
//   &x-amz-server-side-encryption=AES256
//
// NOTE X-Amz-SignedHeaders contains `content-md5` and `x-amz-server-side-encryption` keys as well

const result = await fetch(url, {
  method: 'PUT',
  body: body,
  headers: {
    'Content-Type': 'text/plain',
    'Accept': 'application/json',
    'Content-MD5': bodyMD5, // This is required, otherwise AWS refuses to accept the request
    'x-amz-server-side-encryption': 'AES256' // This is optional, but AWS is okay when value is passed as both query argument and header with same value
  },
});
```

This example shows few issues, which are present when trying to process various presigned requests in localstack, but work correctly in AWS. I try to list them one by one below:

**Issue 1: Allow X-AMZ-Server-Side-Encryption in query string**

Presigned URL query string contains `x-amz-server-side-encryption=AES256` and `X-Amz-Signed-Headers=host%3Bx-amz-server-side-encryption`

PUT request does not contain any extra headers.

Localstack fails to process the request. AWS works.

**Issue 2: Allow X-AMZ-Server-Side-Encryption in query string and PUT request header with same value**

Presigned URL query string contains `x-amz-server-side-encryption=AES256` and `X-Amz-Signed-Headers=host%3Bx-amz-server-side-encryption`

PUT request does contains extra header `x-amz-server-side-encryption=AES256`.

Localstack fails to process the request. AWS works.

**Issue 3: Allow Content-MD5 in query string and PUT request header with same value**

Presigned URL query string contains `Content-MD5=...` and `X-Amz-Signed-Headers=content-md5%3Bhost`

PUT request does contains extra header `Content-MD5=...`.

Localstack fails to process the request. AWS works.

**Issue 4: Allow Content-MD5 and X-AMZ-Server-Side-Encryption in same request**

Combination of issues above. Localstack fails to process the request. AWS works.

### Expected Behavior

_No response_

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

I spin up the localstack docker with arguments below. Just noting that in its simple form, I was able to generate and successfully verify the presigned requests; so it's not an issue of invalid configuration.
```
S3_SKIP_SIGNATURE_VALIDATION=0
AWS_ACCESS_KEY_ID=test
AWS_SECRET_ACCESS_KEY=test`
```

1. Assume you have running Node.js (tested on v12) and Docker.
2. Extract archive [presigned-urls-test.zip](https://github.com/localstack/localstack/files/7863686/presigned-urls-test.zip) and `cd` to this folder
3. `npm install` to install the `aws-sdk` dependency
3. `npm run docker` to spin up the localstack. It will listen on `localhost:4566`
4. `npm run test` to run the test
5. The file should end up with `200` response code in output, but it fails on `403` signature mismatch.


### Environment

```markdown
- OS: Ubuntu 20.04
- Localstack: 0.13.3
- Node.js: 12
- Docker
```


### Anything else?

Signature V4 specification: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html

It seems to me the error is somewhere inside `authenticate_presign_url` method in here

https://github.com/localstack/localstack/blob/a400de0982487df33ae7c7cfc6d255839a03d91e/localstack/services/s3/s3_utils.py#L220-L253

I am not a Python developer, but I have suspicion, that:
* `X-Amz-SignedHeaders` query argument is not parsed to see which extra headers should be included for the signature verification, resulting in `Content-MD5` being skipped
* there is something strange when header with same value is present in both query string and request headers (such as the `Content-MD5` that is neccessary to pass in both, or `X-Amz-Server-Side-Encryption`, that is neccessary to be just in the query string, but AWS allows it to be ALSO in request headers with same value)

Maybe my assumptions are wrong, I was not investigating the code that much.

Hopefully this helps to describe the issue. I know that 


Discussion:
Hi @michal-kocarek, thank you for your patience and such detailed report on your issue, unfortunately I've been unable to reproduce your issue on the latest Localstack version neither on 0.13.3. I tried multiple combinations on the parameters but still I get a successful result. I made sure to create a bucket , then send the PUT request, and the file is there. 

these are the Logs I get:
```
localstack_1  | 2022-03-24T19:09:26:DEBUG:localstack.services.s3.s3_utils: Received presign S3 URL: http://localhost:4566/test-bucket/test-file?Content-MD5=4QrcOUm6Wau%2BVuBX8g%2BIPg%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARCSPW2WNUXC3ZX6P%2F20220324%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220324T190926Z&X-Amz-Expires=3600&X-Amz-Signature=280a188d40a283ee74ff619bf5a780cdf24541eb35e17c1d52c221e118894055&X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-server-side-encryption&x-amz-server-side-encryption=AES256
localstack_1  | 2022-03-24T19:09:26:WARNING:localstack.services.s3.s3_utils: Signatures do not match, but not raising an error, as S3_SKIP_SIGNATURE_VALIDATION=1
localstack_1  | 2022-03-24T19:09:26:DEBUG:localstack.services.s3.s3_utils: Valid presign url.
```

I'm gonna close this issue but if you still have this problem please let know and I'll open the issue again. Also it would very helpful if you provide your logs with your Localstack instance using DEBUG=1. 
Thanks.


Hi @michal-kocarek,
I am currently facing the same problem but with AWS S3.
If you have ever fixed the issue, please let me know, and it's driving me crazy 🤯 

BTW @pinzon, you actually were able to recreate the error, but it was silenced:
`localstack_1  | 2022-03-24T19:09:26:WARNING:localstack.services.s3.s3_utils: Signatures do not match, but not raising an error, as S3_SKIP_SIGNATURE_VALIDATION=1`

I would be grateful for any response 
Hi @myk0la-b , after reading again the issue, it seems I misinterpreted the error. I'll try again as soon as I'm able. Thanks for bringing this to my attention.

Resolving patch:
diff --git a/localstack/services/s3/presigned_url.py b/localstack/services/s3/presigned_url.py
--- a/localstack/services/s3/presigned_url.py
+++ b/localstack/services/s3/presigned_url.py
@@ -589,6 +589,30 @@ def _get_signed_headers_and_filtered_query_string(self) -> Tuple[Dict[str, str],
         headers.pop("Authorization", None)
         signed_headers = self._query_parameters.get("X-Amz-SignedHeaders")
 
+        new_query_args = {}
+        for qs_parameter, qs_value in self._query_parameters.items():
+            # skip the signature
+            if qs_parameter == "X-Amz-Signature":
+                continue
+
+            qs_param_low = qs_parameter.lower()
+            if (
+                qs_parameter not in SIGNATURE_V4_PARAMS
+                and qs_param_low.startswith("x-amz-")
+                and qs_param_low not in headers
+                and qs_param_low in signed_headers
+            ):
+                # AWS JS SDK does not behave like boto, and will add some parameters as query string when signing when
+                # boto would not. this difference in behaviour would lead to pre-signed URLs generated by the JS SDK
+                # to be invalid for the boto signer.
+                # This fixes the behaviour by manually adding the parameter to the headers like boto would, if the SDK
+                # put them in the SignedHeaders
+                # this is especially valid for headers starting with x-amz-server-side-encryption, treated specially in
+                # the old JS SDK
+                headers.add(qs_param_low, qs_value)
+
+            new_query_args[qs_parameter] = qs_value
+
         signature_headers = {}
         not_signed_headers = []
         for header, value in headers.items():
@@ -607,9 +631,6 @@ def _get_signed_headers_and_filtered_query_string(self) -> Tuple[Dict[str, str],
             )
             raise ex
 
-        new_query_args = {
-            arg: value for arg, value in self._query_parameters.items() if arg != "X-Amz-Signature"
-        }
         new_query_string = percent_encode_sequence(new_query_args)
         return signature_headers, new_query_string
 


Test patch:
diff --git a/tests/integration/awslambda/functions/lambda_s3_integration_presign.js b/tests/integration/awslambda/functions/lambda_s3_integration_presign.js
new file mode 100644
--- /dev/null
+++ b/tests/integration/awslambda/functions/lambda_s3_integration_presign.js
@@ -0,0 +1,45 @@
+exports.handler = async (event, context, callback) => {
+    const {
+        S3Client,
+        PutObjectCommand,
+    } = require("@aws-sdk/client-s3");
+    const {
+        getSignedUrl
+    } = require('@aws-sdk/s3-request-presigner');
+
+    const BUCKET_NAME = process.env.AWS_LAMBDA_FUNCTION_NAME;
+    const bodyMd5AsBase64 = '4QrcOUm6Wau+VuBX8g+IPg=='; // body should be '123456'
+
+    let s3;
+    if (process.env.LOCALSTACK_HOSTNAME) {
+        const CREDENTIALS = {
+            secretAccessKey: 'test',
+            accessKeyId: 'test',
+        };
+
+        s3 = new S3Client({
+            endpoint: "http://s3.localhost.localstack.cloud:4566",
+            region: 'us-east-1',
+            credentials: CREDENTIALS,
+        });
+    } else {
+        s3 = new S3Client()
+    }
+
+    const url = await getSignedUrl(
+        s3,
+        new PutObjectCommand({
+            Bucket: BUCKET_NAME,
+            Key: 'key.txt',
+            StorageClass: 'STANDARD',
+            ContentMD5: bodyMd5AsBase64
+        }),
+        {
+            expiresIn: 86400
+        }
+    );
+    return {
+        statusCode: 200,
+        body: JSON.stringify(url)
+    };
+}
diff --git a/tests/integration/awslambda/functions/lambda_s3_integration_sdk_v2.js b/tests/integration/awslambda/functions/lambda_s3_integration_sdk_v2.js
new file mode 100644
--- /dev/null
+++ b/tests/integration/awslambda/functions/lambda_s3_integration_sdk_v2.js
@@ -0,0 +1,47 @@
+exports.handler = async (event, context, callback) => {
+    const AWS = require('aws-sdk');
+
+    const BUCKET_NAME = process.env.AWS_LAMBDA_FUNCTION_NAME;
+    const bodyMd5AsBase64 = '4QrcOUm6Wau+VuBX8g+IPg=='; // body should be '123456'
+
+    let s3;
+    if (process.env.LOCALSTACK_HOSTNAME) {
+        const CREDENTIALS = {
+            secretAccessKey: 'test',
+            accessKeyId: 'test',
+        };
+        s3 = new AWS.S3({
+            endpoint: "http://s3.localhost.localstack.cloud:4566",
+            region: 'us-east-1',
+            signatureVersion: 'v4', // Required for the presigned URL functionality with extra headers
+            credentials: CREDENTIALS
+          });
+
+    } else {
+        s3 = new AWS.S3()
+    }
+
+  const url = s3.getSignedUrl('putObject', {
+    Bucket: BUCKET_NAME,
+    Key: 'key-for-signed-headers-in-qs',
+    Expires: 3600,
+    ServerSideEncryption: 'AES256', // Adds 'X-Amz-Server-Side-Encryption' in query string
+    ContentMD5: bodyMd5AsBase64             // Adds 'Content-MD5' parameter in query string
+    });
+
+    // url: http://localhost:4566/test-bucket-ls-presigned/key-for-signed-headers-in-qs
+    //   ?Content-MD5=4QrcOUm6Wau%2BVuBX8g%2BIPg%3D%3D
+    //   &X-Amz-Algorithm=AWS4-HMAC-SHA256
+    //   &X-Amz-Credential=test%2F20220113%2Fus-east-1%2Fs3%2Faws4_request
+    //   &X-Amz-Date=20220113T142952Z
+    //   &X-Amz-Expires=3600
+    //   &X-Amz-Signature=d219a729f06e37d40a136bb5fec777265b1b34e879f9e338d385b39a3760a14f
+    //   &X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-server-side-encryption
+    //   &x-amz-server-side-encryption=AES256
+    //
+    // NOTE X-Amz-SignedHeaders contains `content-md5` and `x-amz-server-side-encryption` keys as well
+    return {
+        statusCode: 200,
+        body: JSON.stringify(url)
+    };
+}
diff --git a/tests/integration/s3/test_s3.py b/tests/integration/s3/test_s3.py
--- a/tests/integration/s3/test_s3.py
+++ b/tests/integration/s3/test_s3.py
@@ -4260,6 +4260,144 @@ def test_presigned_url_signature_authentication_multi_part(
         assert response.status_code == 200
         assert response.content == data
 
+    @pytest.mark.aws_validated
+    def test_presigned_url_v4_x_amz_in_qs(
+        self,
+        s3_client,
+        s3_bucket,
+        s3_create_bucket,
+        patch_s3_skip_signature_validation_false,
+        lambda_client,
+        create_lambda_function,
+        lambda_su_role,
+        create_tmp_folder_lambda,
+    ):
+        # test that Boto does not hoist x-amz-storage-class in the query string while pre-signing
+        object_key = "temp.txt"
+        client = _s3_client_custom_config(
+            Config(signature_version="s3v4"),
+        )
+        url = client.generate_presigned_url(
+            "put_object",
+            Params={"Bucket": s3_bucket, "Key": object_key, "StorageClass": StorageClass.STANDARD},
+        )
+        assert StorageClass.STANDARD not in url
+
+        handler_file = os.path.join(
+            os.path.dirname(__file__), "../awslambda/functions/lambda_s3_integration_presign.js"
+        )
+        temp_folder = create_tmp_folder_lambda(
+            handler_file,
+            run_command="npm i @aws-sdk/util-endpoints @aws-sdk/client-s3 @aws-sdk/s3-request-presigner @aws-sdk/middleware-endpoint",
+        )
+
+        function_name = f"func-integration-{short_uid()}"
+        create_lambda_function(
+            func_name=function_name,
+            zip_file=testutil.create_zip_file(temp_folder, get_content=True),
+            runtime=LAMBDA_RUNTIME_NODEJS14X,
+            handler="lambda_s3_integration_presign.handler",
+            role=lambda_su_role,
+        )
+        s3_create_bucket(Bucket=function_name)
+
+        response = lambda_client.invoke(FunctionName=function_name)
+        presigned_url = response["Payload"].read()
+        presigned_url = json.loads(to_str(presigned_url))["body"].strip('"')
+        assert StorageClass.STANDARD in presigned_url
+
+        # missing Content-MD5
+        response = requests.put(presigned_url, verify=False, data=b"123456")
+        assert response.status_code == 403
+
+        # AWS needs the Content-MD5 header to validate the integrity of the file as set in the pre-signed URL
+        # but do not provide StorageClass in the headers, because it's not in SignedHeaders
+        response = requests.put(
+            presigned_url,
+            data=b"123456",
+            verify=False,
+            headers={"Content-MD5": "4QrcOUm6Wau+VuBX8g+IPg=="},
+        )
+        assert response.status_code == 200
+
+    @pytest.mark.aws_validated
+    @pytest.mark.skipif(
+        condition=is_old_provider(),
+        reason="behaviour not properly implemented in the legacy provider",
+    )
+    def test_presigned_url_v4_signed_headers_in_qs(
+        self,
+        s3_client,
+        s3_bucket,
+        s3_create_bucket,
+        patch_s3_skip_signature_validation_false,
+        lambda_client,
+        create_lambda_function,
+        lambda_su_role,
+        create_tmp_folder_lambda,
+    ):
+        # test that Boto does not hoist x-amz-server-side-encryption in the query string while pre-signing
+        # it means we would need to provide it in the request headers
+        object_key = "temp.txt"
+        client = _s3_client_custom_config(
+            Config(signature_version="s3v4"),
+        )
+        url = client.generate_presigned_url(
+            "put_object",
+            Params={"Bucket": s3_bucket, "Key": object_key, "ServerSideEncryption": "AES256"},
+        )
+        assert "=AES256" not in url
+
+        handler_file = os.path.join(
+            os.path.dirname(__file__), "../awslambda/functions/lambda_s3_integration_sdk_v2.js"
+        )
+        temp_folder = create_tmp_folder_lambda(
+            handler_file,
+            run_command="npm i @aws-sdk/util-endpoints @aws-sdk/client-s3 @aws-sdk/s3-request-presigner @aws-sdk/middleware-endpoint",
+        )
+
+        function_name = f"func-integration-{short_uid()}"
+        create_lambda_function(
+            func_name=function_name,
+            zip_file=testutil.create_zip_file(temp_folder, get_content=True),
+            runtime=LAMBDA_RUNTIME_NODEJS14X,
+            handler="lambda_s3_integration_sdk_v2.handler",
+            role=lambda_su_role,
+        )
+        s3_create_bucket(Bucket=function_name)
+
+        response = lambda_client.invoke(FunctionName=function_name)
+        presigned_url = response["Payload"].read()
+        presigned_url = json.loads(to_str(presigned_url))["body"].strip('"')
+        assert "=AES256" in presigned_url
+
+        # AWS needs the Content-MD5 header to validate the integrity of the file as set in the pre-signed URL
+        response = requests.put(presigned_url, verify=False, data=b"123456")
+        assert response.status_code == 403
+
+        # assert that we don't need to give x-amz-server-side-encryption even though it's in SignedHeaders,
+        # because it's in the query string
+        response = requests.put(
+            presigned_url,
+            data=b"123456",
+            verify=False,
+            headers={"Content-MD5": "4QrcOUm6Wau+VuBX8g+IPg=="},
+        )
+        assert response.status_code == 200
+
+        # assert that even if we give x-amz-server-side-encryption, as long as it's the same value as the query string,
+        # it will work
+        response = requests.put(
+            presigned_url,
+            data=b"123456",
+            verify=False,
+            headers={
+                "Content-MD5": "4QrcOUm6Wau+VuBX8g+IPg==",
+                "x-amz-server-side-encryption": "AES256",
+            },
+        )
+        assert response.status_code == 200
+
     @staticmethod
     def _get_presigned_snapshot_transformers(snapshot):
         return [