Repository: localstack/localstack

Issue:
bug: LocalStack.NET all S3 operations fails on LocalStack 2.2
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Hello, I'm the maintainer of the [LocalStack.NET](https://github.com/localstack-dotnet/localstack-dotnet-client) client library. LocalStack.NET is a thin wrapper around the official [aws-sdk-net](https://github.com/aws/aws-sdk-net) just like [localstack-python-client](https://github.com/localstack/localstack-python-client).

I've discovered issues related to any S3-related operation when testing against LocalStack version 2.2. Notably, these operations are successful with previous versions like 1.3.1 and 2.0.

Related Issue: https://github.com/localstack-dotnet/localstack-dotnet-client/issues/24

**Description of the Problem:** The operation fails when attempting any S3-related operation using the LocalStack.NET library. I think the logs provided below might offer more insight.

## Test Scenario

**Scenario Link:** [S3Service_Should_Create_A_Bucket](https://github.com/localstack-dotnet/localstack-dotnet-client/blob/v1.4.1/tests/LocalStack.Client.Functional.Tests/Scenarios/S3/BaseS3Scenario.cs#L18)

```csharp
[Fact]
public async Task S3Service_Should_Create_A_Bucket()
{
    var bucketName = Guid.NewGuid().ToString();
    PutBucketResponse putBucketResponse = await CreateTestBucket(bucketName);

    Assert.Equal(HttpStatusCode.OK, putBucketResponse.HttpStatusCode);
}

protected Task<PutBucketResponse> CreateTestBucket(string bucketName = null)
{
    var putBucketRequest = new PutBucketRequest { BucketName = bucketName ?? BucketName, UseClientRegion = true };

    return AmazonS3.PutBucketAsync(putBucketRequest);
}
```

The test scenario above generates a GUID and uses it as the bucket name when creating a bucket. The above is just an example. All other S3-related operations also fail.

## LocalStack Testcontainer setup
```csharp
public static LocalStackBuilder LocalStackBuilder(string version)
{
    return new LocalStackBuilder().WithImage($"localstack/localstack:{version}")
                                  .WithName($"localStack-{version}-{Guid.NewGuid().ToString().ToLower()}")
                                  .WithEnvironment("DOCKER_HOST", "unix:///var/run/docker.sock")
                                  .WithEnvironment("DEBUG", "1")
                                  .WithEnvironment("LS_LOG", "trace-internal")
                                  .WithPortBinding(4566, true)
                                  .WithCleanUp(true);
}
```

## Mitmproxy raw request

```http
PUT http://s3.eu-central-1.amazonaws.com/d9137298-3452-477d-9c0c-30f7ec52f120/ HTTP/1.1
User-Agent: aws-sdk-dotnet-coreclr/3.7.102.0 aws-sdk-dotnet-core/3.7.200.17 .NET_Core/7.0.9 OS/Microsoft_Windows_10.0.22621 ClientAsync
amz-sdk-invocation-id: 148e1966-07dd-403e-bcb1-b3ea27097db6
amz-sdk-request: attempt=1; max=5
x-amz-security-token: my-AwsSessionToken
Host: s3.eu-central-1.amazonaws.com
X-Amz-Date: 20230817T074856Z
X-Amz-Content-SHA256: 270d7010e28541d025cba79779722247d464e2edd8b448b42fdf618a477e0432
Authorization: AWS4-HMAC-SHA256 Credential=my-AwsAccessKeyId/20230817/eu-central-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=280f9675d5de93f8e991d81b1f0796d70ad031d4d777027398662d48ac7db162
Content-Length: 156
Content-Type: application/xml
Content-MD5: 3TSjRZmsSt7Rb6d4WVpfOQ==

<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>eu-central-1</LocationConstraint></CreateBucketConfiguration>
```

## Localstack container logs

```bash
2023-08-17T07:48:54.135648905Z 2023-08-17T07:48:54.135 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : PUT s3.eu-central-1.amazonaws.comhttp://s3.eu-central-1.amazonaws.com/6d34cfe9-0400-494e-9551-dfaf9798030e/
2023-08-17T07:48:54.140016928Z 2023-08-17T07:48:54.136 ERROR --- [   asgi_gw_0] l.aws.handlers.logging     : exception during call chain
2023-08-17T07:48:54.140025268Z Traceback (most recent call last):
2023-08-17T07:48:54.140027608Z   File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle
2023-08-17T07:48:54.140029348Z     handler(self, self.context, response)
2023-08-17T07:48:54.140030938Z   File "/opt/code/localstack/localstack/aws/handlers/routes.py", line 27, in __call__
2023-08-17T07:48:54.140032538Z     router_response = self.router.dispatch(context.request)
2023-08-17T07:48:54.140033908Z   File "/opt/code/localstack/localstack/http/router.py", line 443, in dispatch
2023-08-17T07:48:54.140035318Z     handler, args = matcher.match(get_raw_path(request), method=request.method)
2023-08-17T07:48:54.140036718Z   File "/opt/code/localstack/.venv/lib/python3.10/site-packages/werkzeug/routing/map.py", line 635, in match
2023-08-17T07:48:54.140038268Z     raise RequestRedirect(
2023-08-17T07:48:54.140039648Z werkzeug.routing.exceptions.RequestRedirect: 308 Permanent Redirect: http://s3.eu-central-1.amazonaws.com/http:/s3.eu-central-1.amazonaws.com/6d34cfe9-0400-494e-9551-dfaf9798030e/
2023-08-17T07:48:54.140233396Z 2023-08-17T07:48:54.140 DEBUG --- [   asgi_gw_0] l.aws.protocol.serializer  : No accept header given. Using request's Content-Type (application/xml) as preferred response Content-Type.
2023-08-17T07:48:54.142987739Z 2023-08-17T07:48:54.142  INFO --- [   asgi_gw_0] localstack.request.http    : PUT /6d34cfe9-0400-494e-9551-dfaf9798030e/ => 500; Request(b'<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>eu-central-1</LocationConstraint></CreateBucketConfiguration>', headers={'User-Agent': 'aws-sdk-dotnet-coreclr/3.7.102.0 aws-sdk-dotnet-core/3.7.200.17 .NET_Core/7.0.9 OS/Microsoft_Windows_10.0.22621 ClientAsync', 'amz-sdk-invocation-id': '148e1966-07dd-403e-bcb1-b3ea27097db6', 'amz-sdk-request': 'attempt=1; max=5', 'x-amz-security-token': 'my-AwsSessionToken', 'Host': 's3.eu-central-1.amazonaws.com', 'X-Amz-Date': '20230817T074851Z', 'X-Amz-Content-SHA256': '270d7010e28541d025cba79779722247d464e2edd8b448b42fdf618a477e0432', 'Authorization': 'AWS4-HMAC-SHA256 Credential=my-AwsAccessKeyId/20230817/eu-central-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=64a84fae5e3cc966e56f4561053b2d74c1893dcc27347eeda8c02c274b8d5bbd', 'Content-Length': '156', 'Content-Type': 'application/xml', 'Content-MD5': '3TSjRZmsSt7Rb6d4WVpfOQ==', 'x-localstack-tgt-api': 's3'}); Response(b'<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<Error><Code>InternalError</Code><Message>exception while calling s3 with unknown operation: Traceback (most recent call last):\n  File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle\n    handler(self, self.context, response)\n  File "/opt/code/localstack/localstack/aws/handlers/routes.py", line 27, in __call__\n    router_response = self.router.dispatch(context.request)\n  File "/opt/code/localstack/localstack/http/router.py", line 443, in dispatch\n    handler, args = matcher.match(get_raw_path(request), method=request.method)\n  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/werkzeug/routing/map.py", line 635, in match\n    raise RequestRedirect(\nwerkzeug.routing.exceptions.RequestRedirect: 308 Permanent Redirect: http://s3.eu-central-1.amazonaws.com/http:/s3.eu-central-1.amazonaws.com/6d34cfe9-0400-494e-9551-dfaf9798030e/\n</Message><RequestId>d5ced6d2-d015-4c6c-8c83-0fef33501c7a</RequestId></Error>', headers={'Content-Type': 'application/xml', 'Content-Length': '981', 'x-amz-request-id': 'd5ced6d2-d015-4c6c-8c83-0fef33501c7a', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
2023-08-17T07:48:54.584494830Z 2023-08-17T07:48:54.584 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : PUT s3.eu-central-1.amazonaws.comhttp://s3.eu-central-1.amazonaws.com/6d34cfe9-0400-494e-9551-dfaf9798030e/
2023-08-17T07:48:54.584981886Z 2023-08-17T07:48:54.584 ERROR --- [   asgi_gw_0] l.aws.handlers.logging     : exception during call chain
2023-08-17T07:48:54.584999376Z Traceback (most recent call last):
2023-08-17T07:48:54.585002226Z   File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle
2023-08-17T07:48:54.585004066Z     handler(self, self.context, response)
2023-08-17T07:48:54.585005506Z   File "/opt/code/localstack/localstack/aws/handlers/routes.py", line 27, in __call__
2023-08-17T07:48:54.585006916Z     router_response = self.router.dispatch(context.request)
2023-08-17T07:48:54.585008256Z   File "/opt/code/localstack/localstack/http/router.py", line 443, in dispatch
2023-08-17T07:48:54.585009746Z     handler, args = matcher.match(get_raw_path(request), method=request.method)
2023-08-17T07:48:54.585011126Z   File "/opt/code/localstack/.venv/lib/python3.10/site-packages/werkzeug/routing/map.py", line 635, in match
2023-08-17T07:48:54.585012576Z     raise RequestRedirect(
2023-08-17T07:48:54.585013906Z werkzeug.routing.exceptions.RequestRedirect: 308 Permanent Redirect: http://s3.eu-central-1.amazonaws.com/http:/s3.eu-central-1.amazonaws.com/6d34cfe9-0400-494e-9551-dfaf9798030e/
2023-08-17T07:48:54.585077746Z 2023-08-17T07:48:54.584 DEBUG --- [   asgi_gw_0] l.aws.protocol.serializer  : No accept header given. Using request's Content-Type (application/xml) as preferred response Content-Type.
2023-08-17T07:48:54.585715972Z 2023-08-17T07:48:54.585  INFO --- [   asgi_gw_0] localstack.request.http    : PUT /6d34cfe9-0400-494e-9551-dfaf9798030e/ => 500; Request(b'<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>eu-central-1</LocationConstraint></CreateBucketConfiguration>', headers={'User-Agent': 'aws-sdk-dotnet-coreclr/3.7.102.0 aws-sdk-dotnet-core/3.7.200.17 .NET_Core/7.0.9 OS/Microsoft_Windows_10.0.22621 ClientAsync', 'amz-sdk-invocation-id': '148e1966-07dd-403e-bcb1-b3ea27097db6', 'amz-sdk-request': 'attempt=2; max=5', 'x-amz-security-token': 'my-AwsSessionToken', 'Host': 's3.eu-central-1.amazonaws.com', 'X-Amz-Date': '20230817T074854Z', 'X-Amz-Content-SHA256': '270d7010e28541d025cba79779722247d464e2edd8b448b42fdf618a477e0432', 'Authorization': 'AWS4-HMAC-SHA256 Credential=my-AwsAccessKeyId/20230817/eu-central-1/s3/aws4_request, SignedHeaders=content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=8bf2a988e09b2cf55f7eace8c036f9858ec742f7cf1b9ce6f3e3be16a4d769a6', 'Content-Length': '156', 'Content-Type': 'application/xml', 'Content-MD5': '3TSjRZmsSt7Rb6d4WVpfOQ==', 'x-localstack-tgt-api': 's3'}); Response(b'<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<Error><Code>InternalError</Code><Message>exception while calling s3 with unknown operation: Traceback (most recent call last):\n  File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle\n    handler(self, self.context, response)\n  File "/opt/code/localstack/localstack/aws/handlers/routes.py", line 27, in __call__\n    router_response = self.router.dispatch(context.request)\n  File "/opt/code/localstack/localstack/http/router.py", line 443, in dispatch\n    handler, args = matcher.match(get_raw_path(request), method=request.method)\n  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/werkzeug/routing/map.py", line 635, in match\n    raise RequestRedirect(\nwerkzeug.routing.exceptions.RequestRedirect: 308 Permanent Redirect: http://s3.eu-central-1.amazonaws.com/http:/s3.eu-central-1.amazonaws.com/6d34cfe9-0400-494e-9551-dfaf9798030e/\n</Message><RequestId>11e660d6-ff95-490b-9e59-4af093e60738</RequestId></Error>', headers={'Content-Type': 'application/xml', 'Content-Length': '981', 'x-amz-request-id': '11e660d6-ff95-490b-9e59-4af093e60738', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=', 'Connection': 'close'})
```


### Expected Behavior

Using the LocalStack.NET library, all S3-related operations should be successful, consistent with the behavior observed on LocalStack versions 1.3.1 and 2.0.

### How are you starting LocalStack?

Custom (please describe below)

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

Using TestContainers with the specific configurations mentioned above.

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Using the [LocalStack.NET](https://github.com/localstack-dotnet/localstack-dotnet-client) client library mentioned above. I can provide and example project for this specific case as well.


### Environment

```markdown
- OS: Windows 11 x64
- LocalStack: 2.2
- Docker: Docker Desktop running on WSL 2
```


### Anything else?

_No response_


Discussion:
Hello @Blind-Striker and thanks for your report.

I'm a bit confused by the log: this what the request looks like, and this does not look like a valid URL to me? `s3.eu-central-1.amazonaws.comhttp://s3.eu-central-1.amazonaws.com/6d34cfe9-0400-494e-9551-dfaf9798030e/` ? How is this URL being constructed?
And when targeting LocalStack 2.0, what does this URL looks like? Also, how is the request targeting `s3.eu-central-1.amazonaws.com` routed to LocalStack? There might have been change I'm not aware in the routing. Also, does this affect only S3? 
Hey @bentsku,

Thank you for getting back so swiftly.

> Also, does this affect only S3?

No. There is also a similar issue about SQS I opened yesterday: https://github.com/localstack/localstack/issues/8924

> I'm a bit confused by the log: this what the request looks like, and this does not look like a valid URL to me? s3.eu-central-1.amazonaws.comhttp://s3.eu-central-1.amazonaws.com/6d34cfe9-0400-494e-9551-dfaf9798030e/ ?

Yes, I noticed that too, but to be sure, I set up a proxy container to see my actual HTTP request, which is below;

```http
PUT http://s3.eu-central-1.amazonaws.com/d9137298-3452-477d-9c0c-30f7ec52f120/ HTTP/1.1
User-Agent: aws-sdk-dotnet-coreclr/3.7.102.0 aws-sdk-dotnet-core/3.7.200.17 .NET_Core/7.0.9 OS/Microsoft_Windows_10.0.22621 ClientAsync
amz-sdk-invocation-id: 148e1966-07dd-403e-bcb1-b3ea27097db6
amz-sdk-request: attempt=1; max=5
x-amz-security-token: my-AwsSessionToken
Host: s3.eu-central-1.amazonaws.com
X-Amz-Date: 20230817T074856Z
X-Amz-Content-SHA256: 270d7010e28541d025cba79779722247d464e2edd8b448b42fdf618a477e0432
Authorization: AWS4-HMAC-SHA256 Credential=my-AwsAccessKeyId/20230817/eu-central-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=280f9675d5de93f8e991d81b1f0796d70ad031d4d777027398662d48ac7db162
Content-Length: 156
Content-Type: application/xml
Content-MD5: 3TSjRZmsSt7Rb6d4WVpfOQ==

<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>eu-central-1</LocationConstraint></CreateBucketConfiguration>
```

This is precisely what the LocalStack.NET (via AWS SDK .NET) sends to the Localstack container.

> How is this URL being constructed? Also, how is the request targeting s3.eu-central-1.amazonaws.com routed to LocalStack?

LocalStack.NET is a thin wrapper around the official [aws-sdk-net](https://github.com/aws/aws-sdk-net) just like [localstack-python-client](https://github.com/localstack/localstack-python-client). Essentially, LocalStack.NET intercepts the creation of AWS clients (such as S3, SNS, SQS, and more) and adjusts specific properties. For each client, it sets the `ProxyHost` and `ProxyPort` properties to the localstack host and the edge port. Unless a user explicitly configures `useServiceUrl` to true, the `RegionEndpoint` is consistently used. The AWS SDK.NET Client Config possesses two properties: `ServiceUrl` and `RegionEndpoint`. These properties are mutually exclusive. When one property is adjusted, the other is automatically reset to null. LocalStack.NET has prioritized the `RegionEndpoint` property, a setup that worked seamlessly up to LocalStack v2.2

>How s3.eu-central-1.amazonaws.com routed to LocalStack?

The likely reason is my setting of the `ProxyHost` and `ProxyPort` properties. The AWS SDK.NET Client is responsible for generating the URL. Each AWS client has an associated endpoint provider. For instance, the endpoint provider for S3 can be found at:

[AmazonS3EndpointProvider](https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Services/S3/Generated/Internal/AmazonS3EndpointProvider.cs)

While it's possible to customize this, the existing endpoint provider was compatible with LocalStack up until version v2.2. Personally, I'd rather avoid altering endpoint providers if that's the root of the problem. 😄

> And when targeting LocalStack 2.0, what does this URL look like?

Here it is;

```bash
2023-08-17T10:34:05.607 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : PUT s3.eu-central-1.amazonaws.comhttp://s3.eu-central-1.amazonaws.com/7c75cab3-964b-4278-8e9f-e6e34f2192bf/
2023-08-17T10:34:05.608083696Z 2023-08-17T10:34:05.607 DEBUG --- [   asgi_gw_0] l.services.s3.virtual_host : Rewritten original host url: http://s3.eu-central-1.amazonaws.com/7c75cab3-964b-4278-8e9f-e6e34f2192bf/ to path-style url: http://s3.localhost.localstack.cloud:4566/7c75cab3-964b-4278-8e9f-e6e34f2192bf/
2023-08-17T10:34:05.611631354Z 2023-08-17T10:34:05.611 DEBUG --- [   asgi_gw_1] l.aws.serving.wsgi         : PUT s3.localhost.localstack.cloud:4566/7c75cab3-964b-4278-8e9f-e6e34f2192bf/
2023-08-17T10:34:05.621919540Z 2023-08-17T10:34:05.621 DEBUG --- [   asgi_gw_1] plugin.manager             : instantiating plugin PluginSpec(localstack.aws.provider.s3:default = <function s3 at 0x7f6abe7b36d0>)
2023-08-17T10:34:05.621946450Z 2023-08-17T10:34:05.621 DEBUG --- [   asgi_gw_1] plugin.manager             : loading plugin localstack.aws.provider:s3:default
2023-08-17T10:34:05.655702064Z 2023-08-17T10:34:05.655 DEBUG --- [   asgi_gw_1] l.services.plugins         : checking service health s3:4566
2023-08-17T10:34:05.711526754Z 2023-08-17T10:34:05.711 DEBUG --- [   asgi_gw_1] l.aws.protocol.serializer  : No accept header given. Using request's Content-Type (application/xml) as preferred response Content-Type.
2023-08-17T10:34:05.712246366Z 2023-08-17T10:34:05.711  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.CreateBucket => 200; CreateBucketRequest({'ACL': None, 'Bucket': '7c75cab3-964b-4278-8e9f-e6e34f2192bf', 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWrite': None, 'GrantWriteACP': None, 'ObjectLockEnabledForBucket': None, 'ObjectOwnership': None, 'CreateBucketConfiguration': {'LocationConstraint': 'eu-central-1'}}, headers={'User-Agent': 'aws-sdk-dotnet-coreclr/3.7.102.0 aws-sdk-dotnet-core/3.7.200.17 .NET_Core/7.0.9 OS/Microsoft_Windows_10.0.22621 ClientAsync', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Host': 's3.localhost.localstack.cloud:4566', 'Content-Type': 'application/xml', 'Content-Length': '156', 'Amz-Sdk-Invocation-Id': '7e45bd84-a058-4094-bd08-3671f71c14c2', 'Amz-Sdk-Request': 'attempt=1; max=5', 'X-Amz-Security-Token': 'my-AwsSessionToken', 'X-Amz-Date': '20230817T103346Z', 'X-Amz-Content-Sha256': '270d7010e28541d025cba79779722247d464e2edd8b448b42fdf618a477e0432', 'Authorization': 'AWS4-HMAC-SHA256 Credential=my-AwsAccessKeyId/20230817/eu-central-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=7d66315949baa9d893956de40979318ab57d36c2b5248b5bd6506632d5e4ba2b', 'Content-Md5': '3TSjRZmsSt7Rb6d4WVpfOQ==', 'x-localstack-tgt-api': 's3', 'X-S3-Vhost-Forwarded-For': 's3.eu-central-1.amazonaws.com', 'X-Forwarded-For': '172.17.0.1', 'x-moto-account-id': '000000000000'}); CreateBucketOutput({'Location': 'http://7c75cab3-964b-4278-8e9f-e6e34f2192bf.s3.localhost.localstack.cloud:4566/'}, headers={'Content-Type': 'application/xml', 'Location': 'http://7c75cab3-964b-4278-8e9f-e6e34f2192bf.s3.localhost.localstack.cloud:4566/', 'Content-Length': '61', 'x-amz-request-id': 'X61SFMMCU5TA82XB29WLUFMN9BZKFH3C20XYQ0MQPFPG8S2DQDAH', 'x-amz-id-2': 'MzRISOwyjmnupX61SFMMCU5TA82XB29WLUFMN9BZKFH3C20XYQ0MQPFPG8S2DQDAH7/JypPGXLh0OVFGcJaaO3KW/hRAqKOpIEEp'})
2023-08-17T10:34:05.714574847Z 2023-08-17T10:34:05.714  INFO --- [   asgi_gw_0] localstack.request.http    : PUT /7c75cab3-964b-4278-8e9f-e6e34f2192bf/ => 200; Request(b'<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>eu-central-1</LocationConstraint></CreateBucketConfiguration>', headers={'User-Agent': 'aws-sdk-dotnet-coreclr/3.7.102.0 aws-sdk-dotnet-core/3.7.200.17 .NET_Core/7.0.9 OS/Microsoft_Windows_10.0.22621 ClientAsync', 'amz-sdk-invocation-id': '7e45bd84-a058-4094-bd08-3671f71c14c2', 'amz-sdk-request': 'attempt=1; max=5', 'x-amz-security-token': 'my-AwsSessionToken', 'Host': 's3.eu-central-1.amazonaws.com', 'X-Amz-Date': '20230817T103346Z', 'X-Amz-Content-SHA256': '270d7010e28541d025cba79779722247d464e2edd8b448b42fdf618a477e0432', 'Authorization': 'AWS4-HMAC-SHA256 Credential=my-AwsAccessKeyId/20230817/eu-central-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=7d66315949baa9d893956de40979318ab57d36c2b5248b5bd6506632d5e4ba2b', 'Content-Length': '156', 'Content-Type': 'application/xml', 'Content-MD5': '3TSjRZmsSt7Rb6d4WVpfOQ==', 'x-localstack-tgt-api': 's3'}); Response(b"<?xml version='1.0' encoding='utf-8'?>\n<CreateBucketOutput />", headers={'Content-Type': 'application/xml', 'Location': 'http://7c75cab3-964b-4278-8e9f-e6e34f2192bf.s3.localhost.localstack.cloud:4566/', 'Content-Length': '61', 'x-amz-request-id': 'X61SFMMCU5TA82XB29WLUFMN9BZKFH3C20XYQ0MQPFPG8S2DQDAH', 'x-amz-id-2': 'MzRISOwyjmnupX61SFMMCU5TA82XB29WLUFMN9BZKFH3C20XYQ0MQPFPG8S2DQDAH7/JypPGXLh0OVFGcJaaO3KW/hRAqKOpIEEp', 'Connection': 'close'})
2023-08-17T10:34:07.938188778Z 2023-08-17T10:34:07.938 DEBUG --- [   asgi_gw_1] l.aws.serving.wsgi         : GET localhost:4566/_localstack/health
2023-08-17T10:34:18.265179468Z 2023-08-17T10:34:18.264 DEBUG --- [   asgi_gw_0] l.aws.serving.wsgi         : GET localhost:4566/_localstack/health
2023-08-17T10:34:18.265318957Z 2023-08-17T10:34:18.265 DEBUG --- [   asgi_gw_0] localstack.aws.accounts    : No Account ID in thread-local storage for thread 140096305088256
```

The line below caught my attention;

```bash
Rewritten original host url: http://s3.eu-central-1.amazonaws.com/7c75cab3-964b-4278-8e9f-e6e34f2192bf/ to path-style url: http://s3.localhost.localstack.cloud:4566/7c75cab3-964b-4278-8e9f-e6e34f2192bf/
```

Maybe the rewriting logic has been removed in the new version.

I think this might have something to do with some Werkzeug upgrade we had. But anyway, `s3.eu-central-1.amazonaws.comhttp://s3.eu-central-1.amazonaws.com/7c75cab3-964b-4278-8e9f-e6e34f2192bf/` is not a valid Host header, and I think before we accidentally supported it.
Also, the rewriting is not supposed to work for that: is it supposed to match when receiving a request in the format of `http://<bucket>.s3.<domain>/<key?>` to rewrite it to this format `http://s3.<domain>/<bucket>/<key?>`.

I'll need to check exactly how we handle the proxying of request, but I suspect there is something going on with the `Proxy` settings. Also, as the problem is exactly the same with SQS, this is related to the parsing of the `Host` header, and how we match request. 
I'm unsure how the 'Host' header could have been sent incorrectly. As I said, I forwarded my request to a proxy for verification, and I observed the 'Host' header as follows:

```bash
Host: s3.eu-central-1.amazonaws.com
```

I'm not sure if the issue with SQS is identical, especially because, unlike S3, all other operations in SQS are working except for the 'delete' operation.

Thank you again for your support. If you'd like, I can develop a small .NET console application that allows you to test both issues on your local machine.
I'm not sure, but the value we're seeing is from the Host header LocalStack is receiving. Is there an additional layer between your mitm proxy and LocalStack? Because I'm not sure how `s3.eu-central-1.amazonaws.com` is then resolved to LocalStack? I might be missing some knowledge about proxying here, and I could rope in another member of the team. 

The issue in SQS is similar, because from the report, the Host header is also weird: 
`POST sqs.eu-central-1.amazonaws.comhttp://sqs.eu-central-1.amazonaws.com/000000000000/c85c8994-510d-4f6e-801d-4b7753932583.fifo`
It is exactly the same issue it seems.

But yes a small console application could help reproducing the error and maybe see why LocalStack interprets the Host header like that?
> But yes a small console application could help reproducing the error and maybe see why LocalStack interprets the Host header like that?

Ok, give me half an hour max.

> Is there an additional layer between your mitm proxy and LocalStack?

No, there wasn't any additional layer between my app and the container instance expect mitm.

What's interesting is that among my functional tests, I also have tests for DynamoDB and SNS. In addition to these, I have specific real-world scenarios, such as creating an SNS, adding SQS as a consumer, reading the published messages from SQS, and checking the CloudWatch logs. I perform all of these tasks using the SDK, and they work across all versions, including v2.2. I'll then add a few working scenarios to the console application as well.
I think it works for other services because those services don't need to parse the `Host` header / request URL. Perfect, I'll wait for that and give it a try later today! Thanks a lot 🙏 

Hey @bentsku,

I've completed the sample application:

[localstack-s3-sqs-issue-sandbox](https://github.com/Blind-Striker/localstack-s3-sqs-issue-sandbox)

The README provides detailed instructions. If you have any more questions, I'm happy to answer them. Thank you again for your support.
Hey @bentsku 👋, good morning☀️ . Do you have any updates on this issue?
Hello @Blind-Striker! Yes, sorry for coming back so late.

I'm currently investigating, and it seems it could come from the underlying HTTP request and how we parse it.

When doing the request with the CLI:
```
PUT /test HTTP/1.1 
Host: localhost:4566 
Accept-Encoding: identity 
User-Agent: aws-cli/1.24.5 Python/3.9.17 Darwin/22.6.0 botocore/1.26.5 
X-Amz-Date: 20230822T152045Z 
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 
Authorization: AWS4-HMAC-SHA256 Credential=test/20230822/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=630e36a60f1074f5ad7e41cbbf7cf4297fc39491fc762eeadd6bd91850516013 
amz-sdk-invocation-id: 93203783-aff8-4595-a9a6-6fbad76c6a24 
amz-sdk-request: attempt=1 
Content-Length: 0 
```
Note the relative URL when in the HTTP request start line: `PUT /test HTTP/1.1`

Whereas the request coming out of your SDK has a full URL:
```
PUT http://s3.eu-central-1.amazonaws.com/1b38df2b-24e1-4565-ae9e-a957a7681c38/ HTTP/1.1 
User-Agent: aws-sdk-dotnet-coreclr/3.7.201.16 aws-sdk-dotnet-core/3.7.200.18 .NET_Core/7.0.9 OS/Darwin_22.6.0_Darwin_Kernel_Version_22.6.0:_Wed_Jul__5_22:21:56_PDT_2023;_root:xnu-8796.141.3~6/RELEASE_X86_64 ClientAsync 
amz-sdk-invocation-id: 3014b6bf-3d88-4516-8f45-fea3357efe0a 
amz-sdk-request: attempt=1; max=1 
x-amz-security-token: my-AwsSessionToken 
Host: s3.eu-central-1.amazonaws.com 
X-Amz-Date: 20230822T152830Z 
X-Amz-Content-SHA256: 270d7010e28541d025cba79779722247d464e2edd8b448b42fdf618a477e0432 
Authorization: AWS4-HMAC-SHA256 Credential=my-AwsAccessKeyId/20230822/eu-central-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=0391771fe6455cbc97e686ab9bb52954f4152e50cc8324597c8fbc28ff6500ab 
Content-Length: 156 
Content-Type: application/xml 
Content-MD5: 3TSjRZmsSt7Rb6d4WVpfOQ== 
 
<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>eu-central-1</LocationConstraint></CreateBucketConfiguration>
```
Note here: `PUT http://s3.eu-central-1.amazonaws.com/1b38df2b-24e1-4565-ae9e-a957a7681c38/ HTTP/1.1`

It seems something is wrong with that, I'll continue the investigation and come back to you.
It's not really a regression, as it seems we were not handling correctly before either, and there might been some hidden issues (for example when not using `forcePathStyle`). 

Resolving patch:
diff --git a/localstack/http/request.py b/localstack/http/request.py
--- a/localstack/http/request.py
+++ b/localstack/http/request.py
@@ -1,6 +1,6 @@
 from io import BytesIO
 from typing import IO, TYPE_CHECKING, Dict, Mapping, Optional, Tuple, Union
-from urllib.parse import quote, unquote, urlencode
+from urllib.parse import quote, unquote, urlencode, urlparse
 
 if TYPE_CHECKING:
     from _typeshed.wsgi import WSGIEnvironment
@@ -187,15 +187,16 @@ def application(cls, *args):
 def get_raw_path(request) -> str:
     """
     Returns the raw_path inside the request without the query string. The request can either be a Quart Request
-    object (that encodes the raw path in request.scope['raw_path']) or a Werkzeug WSGi request (that encodes the raw
-    path in request.environ['RAW_URI']).
+    object (that encodes the raw path in request.scope['raw_path']) or a Werkzeug WSGI request (that encodes the raw
+    URI in request.environ['RAW_URI']).
 
     :param request: the request object
     :return: the raw path if any
     """
     if hasattr(request, "environ"):
         # werkzeug/flask request (already a string, and contains the query part)
-        return request.environ.get("RAW_URI", request.path).split("?")[0]
+        # we need to parse it, because the RAW_URI can contain a full URL if it is specified in the HTTP request
+        return urlparse(request.environ.get("RAW_URI", request.path)).path
 
     if hasattr(request, "scope"):
         # quart request raw_path comes as bytes, and without the query part


Test patch:
diff --git a/tests/unit/http_/test_request.py b/tests/unit/http_/test_request.py
--- a/tests/unit/http_/test_request.py
+++ b/tests/unit/http_/test_request.py
@@ -137,6 +137,16 @@ def test_get_raw_path_with_query():
     assert get_raw_path(request) == "/foo%2Fbar/ed"
 
 
+def test_get_raw_path_with_full_uri():
+    # raw_path is actually raw_uri in the WSGI environment
+    # it can be a full URL
+    request = Request("GET", "/foo/bar/ed", raw_path="http://localhost:4566/foo%2Fbar/ed")
+
+    assert request.path == "/foo/bar/ed"
+    assert request.environ["RAW_URI"] == "http://localhost:4566/foo%2Fbar/ed"
+    assert get_raw_path(request) == "/foo%2Fbar/ed"
+
+
 def test_headers_retain_dashes():
     request = Request("GET", "/foo/bar/ed", {"X-Amz-Meta--foo_bar-ed": "foobar"})
     assert "x-amz-meta--foo_bar-ed" in request.headers