Repository: localstack/localstack

Issue:
bug: Dummy WSGi Request can't handle non-latin-1 s3 keys
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

With the latest docker image a boto3 `put_object` call with a non-latin1 key results in a 500.

Quick example, with `localstack/localstack:latest` running in docker-compose at the host `localstack`:
```bash
# Attempt to upload file.
$ python upload_test.py
ClientError: An error occurred (500) when calling the PutObject operation (reached max retries: 4): Internal Server Error
```

Stack track from localstack container:
```python
2022-04-22T20:14:21.148:INFO:localstack.utils.threads: Thread run method <function AdaptiveThreadPool.submit.<locals>._run at 0x7f4228aca700>(None) failed: 'latin-1' codec can't encode character '\u0100' in position 29: ordinal not in range(256) Traceback (most recent call last):
  File "/opt/code/localstack/localstack/utils/threads.py", line 38, in run
    result = self.func(self.params, **kwargs)
  File "/opt/code/localstack/localstack/utils/asyncio.py", line 30, in _run
    return fn(*args, **kwargs)
  File "/opt/code/localstack/localstack/services/generic_proxy.py", line 972, in handler
    return modify_and_forward(
  File "/opt/code/localstack/localstack/services/generic_proxy.py", line 516, in wrapper
    value = func(*args, **kwargs)
  File "/opt/code/localstack/localstack/services/generic_proxy.py", line 596, in modify_and_forward
    listener_result = listener.forward_request(
  File "/opt/code/localstack/localstack/services/internal.py", line 219, in forward_request
    return super().forward_request(method, path, data, headers)
  File "/opt/code/localstack/localstack/http/adapters.py", line 63, in forward_request
    return super().forward_request(method, path, data, headers)
  File "/opt/code/localstack/localstack/http/adapters.py", line 29, in forward_request
    request = Request(
  File "/opt/code/localstack/localstack/http/request.py", line 144, in __init__
    super(Request, self).__init__(environ)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/werkzeug/wrappers/request.py", line 120, in __init__
    path=_wsgi_decoding_dance(
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/werkzeug/_internal.py", line 153, in _wsgi_decoding_dance
    return s.encode("latin1").decode(charset, errors)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0100' in position 29: ordinal not in range(256)
```

### Expected Behavior

On version `0.14.2` the `put_object` call completes as expected. File shows up in the bucket normally and can be fetched.

Example with same test file:
```bash
$ python upload_test.py
{'ETag': '"e99a18c428cb38d5f260853678922e03"',
 'ResponseMetadata': {'HTTPHeaders': {'access-control-allow-headers': 'authorization,cache-control,content-length,content-md5,content-type,etag,location,x-amz-acl,x-amz-content-sha256,x-amz-date,x-amz-request-id,x-amz-security-token,x-amz-tagging,x-amz-target,x-amz-user-agent,x-amz-version-id,x-amzn-requestid,x-localstack-target,amz-sdk-invocation-id,amz-sdk-request',
                                      'access-control-allow-methods': 'HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH',
                                      'access-control-allow-origin': '*',
                                      'access-control-expose-headers': 'etag,x-amz-version-id',
                                      'connection': 'close',
                                      'content-length': '0',
                                      'content-type': 'text/html; '
                                                      'charset=utf-8',
                                      'date': 'Fri, 22 Apr 2022 20:34:28 GMT',
                                      'etag': '"e99a18c428cb38d5f260853678922e03"',
                                      'last-modified': 'Fri, 22 Apr 2022 '
                                                       '20:34:28 GMT',
                                      'location': '/testbucket',
                                      'server': 'hypercorn-h11',
                                      'x-amz-id-2': 'MzRISOwyjmnupB3D2CD1CB9895A327/JypPGXLh0OVFGcJaaO3KW/hRAqKOpIEEp',
                                      'x-amz-request-id': 'B3D2CD1CB9895A32',
                                      'x-amzn-requestid': 'Q5CQSHL76V5GF8KJ6V71AKDCFDHRKCTV8WUI00W02WWYNP96GFDJ'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'Q5CQSHL76V5GF8KJ6V71AKDCFDHRKCTV8WUI00W02WWYNP96GFDJ',
                      'RetryAttempts': 0}}
$
$  LOCALSTACK_HOST=localstack awslocal s3 ls testbucket
2022-04-22 20:50:53          6 Ā0
$ LOCALSTACK_HOST=localstack awslocal s3 cp s3://testbucket/Ā0 .
download: s3://testbucket/Ā0 to ./Ā0
$ xxd ./Ā0
00000000: 6162 6331 3233                           abc123
```

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)
```yaml
services:
  localstack:
    # image: localstack/localstack:0.14.2
    image: localstack/localstack:latest
    environment:
      - SERVICES=s3
      - REGION=us-west-2
      - ENDPOINT=http://localstack:4566
    ports:
      - "4566:4566"
  # container to run tests
  develop:
    image: python:3.9
    entrypoint:
      - "tail"
      - "-f"
      - "/dev/null"
    environment:
      - AWS_ACCESS_KEY_ID=foobar
      - AWS_SECRET_ACCESS_KEY=foobar
    depends_on:
      - localstack
```
#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

##### Python Test File
```python
# upload_test.py
from pprint import pprint
import boto3
s3_client = boto3.client('s3', endpoint_url="http://localstack:4566")
pprint(s3_client.put_object(Bucket="testbucket", Key = 'Ā0', Body = b'abc123'))
```
##### Test Session
```bash
# Create new test bucket.
$ LOCALSTACK_HOST=localstack awslocal s3api create-bucket --bucket testbucket
{
    "Location": "/testbucket"
}
# Currently empty.
$ LOCALSTACK_HOST=localstack awslocal s3 ls testbucket
$
# Attempt to upload file.
$ python upload_test.py
Traceback (most recent call last):
  File "//upload_test.py", line 5, in <module>
    pprint(s3_client.put_object(Bucket="testbucket", Key = 'Ā0', Body = b'abc123'))
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 415, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 745, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (500) when calling the PutObject operation (reached max retries: 4): Internal Server Error
```

### Environment

```markdown
- OS: Ubuntu 20.04
- LocalStack: latest
```


### Anything else?

Looks like this started happening after https://github.com/localstack/localstack/pull/5876.


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [developer guide](https://docs.localstack.cloud/developer-guide/).
Thanks for the detailed report Wes @noblepayne. Looks pretty clear what the problem is, I'll poke around and see if I can come up with a quick fix tomorrow.

Resolving patch:
diff --git a/localstack/http/adapters.py b/localstack/http/adapters.py
--- a/localstack/http/adapters.py
+++ b/localstack/http/adapters.py
@@ -8,7 +8,7 @@
 
 from localstack.services.generic_proxy import ProxyListener
 
-from .request import Request
+from .request import Request, get_raw_path
 from .response import Response
 from .router import Router
 
@@ -24,7 +24,7 @@ def request(self, request: Request) -> Response:
 
     def forward_request(self, method, path, data, headers):
         split_url = urlsplit(path)
-        raw_path = quart_request.scope.get("raw_path")
+        raw_path = get_raw_path(quart_request)
 
         request = Request(
             method=method,
diff --git a/localstack/http/request.py b/localstack/http/request.py
--- a/localstack/http/request.py
+++ b/localstack/http/request.py
@@ -1,5 +1,6 @@
 from io import BytesIO
 from typing import TYPE_CHECKING, Dict, Mapping, Optional, Tuple, Union
+from urllib.parse import quote, unquote
 
 if TYPE_CHECKING:
     from _typeshed.wsgi import WSGIEnvironment
@@ -46,8 +47,9 @@ def dummy_wsgi_environment(
     # Standard environ keys
     environ = {
         "REQUEST_METHOD": method,
-        "SCRIPT_NAME": root_path.rstrip("/"),
-        "PATH_INFO": path,
+        # prepare the paths for the "WSGI decoding dance" done by werkzeug
+        "SCRIPT_NAME": unquote(quote(root_path.rstrip("/")), "latin-1"),
+        "PATH_INFO": unquote(quote(path), "latin-1"),
         "SERVER_PROTOCOL": "HTTP/1.1",
     }
 
@@ -158,11 +160,11 @@ def get_raw_path(request) -> str:
     :return: the raw path if any
     """
     if hasattr(request, "environ"):
-        # werkzeug/flask request
+        # werkzeug/flask request (already a string)
         return request.environ.get("RAW_URI", request.path)
 
     if hasattr(request, "scope"):
-        # quart request
-        return request.scope.get("raw_path", request.path)
+        # quart request raw_path comes as bytes
+        return request.scope.get("raw_path", request.path).decode("utf-8")
 
     raise ValueError("cannot extract raw path from request object %s" % request)


Test patch:
diff --git a/tests/integration/test_s3.py b/tests/integration/test_s3.py
--- a/tests/integration/test_s3.py
+++ b/tests/integration/test_s3.py
@@ -2706,6 +2706,14 @@ def test_delete_bucket_with_content(self, s3_client, s3_create_bucket):
         resp = s3_client.list_buckets()
         assert len(resp["Buckets"]) == 0
 
+    @pytest.mark.aws_validated
+    def test_put_and_get_object_with_utf8_key(self, s3_client, s3_create_bucket):
+        bucket = s3_create_bucket()
+        response = s3_client.put_object(Bucket=bucket, Key="Ā0Ä", Body=b"abc123")
+        assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
+        response = s3_client.get_object(Bucket=bucket, Key="Ā0Ä")
+        assert response["Body"].read() == b"abc123"
+
 
 @pytest.mark.parametrize(
     "api_version, bucket_name, payload",
diff --git a/tests/unit/aws/protocol/test_parser.py b/tests/unit/aws/protocol/test_parser.py
--- a/tests/unit/aws/protocol/test_parser.py
+++ b/tests/unit/aws/protocol/test_parser.py
@@ -1,5 +1,5 @@
 from datetime import datetime, timezone
-from urllib.parse import unquote, urlencode, urlsplit
+from urllib.parse import urlencode, urlsplit
 
 import pytest
 from botocore.awsrequest import prepare_request_dict
@@ -251,7 +251,7 @@ def _botocore_parser_integration_test(
     parsed_operation_model, parsed_request = parser.parse(
         HttpRequest(
             method=serialized_request.get("method") or "GET",
-            path=unquote(path),
+            path=path,
             query_string=to_str(query_string),
             headers=headers,
             body=body,
@@ -754,6 +754,18 @@ def test_parse_s3_with_extended_uri_pattern():
     )
 
 
+def test_parse_s3_utf8_url():
+    """Test the parsing of a map with the location trait 'headers'."""
+    _botocore_parser_integration_test(
+        service="s3",
+        action="PutObject",
+        ContentLength=0,
+        Bucket="test-bucket",
+        Key="Ā0",
+        Metadata={"Key": "value", "Key2": "value2"},
+    )
+
+
 def test_parse_restjson_uri_location():
     """Tests if the parsing of uri parameters works correctly for the rest-json protocol"""
     _botocore_parser_integration_test(