Repository: sanic-org/sanic

Issue:
HTTP 1 request headers decoded using default encoding instead of ISO-8859-1
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

headers are decoded here without specifying their encoding:

https://github.com/sanic-org/sanic/blob/ad4e526c775fc3ce950503d6476d9d344492b0dd/sanic/http/http1.py#L205

On my system (osx using python 3.10.8 installed via homebrew) this causes bytes that are valid characters in ISO-8859-1 but not in UTF-8 to be decoded as surrogate escape characters, e.g. `b"\x80"` becomes `"\udf80"` instead of `"\x80"`

### Code snippet

_No response_

### Expected Behavior

headers encoded as ISO-8859-1 with no MIME type to be decoded correctly without using UTF-8 surrogate escape characters.

### How do you run Sanic?

As a script (`app.run` or `Sanic.serve`)

### Operating System

linux

### Sanic Version

22.9.1

### Additional context

this used to work as expected in Sanic<=20.12.7


Discussion:
Which client are you using? Browsers should generally stick to ASCII in request headers but will recognize UTF-8 in responses. Using anything other than ASCII is discouraged, and servers should handle it as opaque binary (not ISO-8859-1, and you cannot specify the charset used in headers). IIRC, you *can* indeed supply binary values in browsers using Javascript `fetch` API and there is nothing to say which charset it is in.

Sanic uses surrogate escapes so that you *can* encode it back to original binary, and then decode as ISO-8859-1 if you wish, but I would strongly advice updating your systems to use UTF-8 or ASCII if in any way possible.
> I would strongly advice updating your systems to use UTF-8 or ASCII if in any way possible.

my clients code is supposed to only send ASCII, but unfortunately I don't own the client code, in practice my application receives garbage that needs to be preserved, and changing how it handles that garbage is undesirable.

> you _can_ encode it back to original binary, and then decode as ISO-8859-1 if you wish

I had not figured this out, and that's sufficient for my use case, thank you.
Earlier Sanic versions handled headers as ISO-8859-1, which was causing trouble when they actually were in UTF-8 (more common nowadays). I had to put a lot of thought into this while reimplementing the HTTP parser code as leaving them as `bytes` wouldn't be practical either. The surrogate escape coding is [WTF-8](https://simonsapin.github.io/wtf-8/) which indeed is meant for *preserving garbage*, being able to restore original bytes of what might be ill-formed UTF-8. I'm glad you found use for this detail of Sanic's implementation, being able to restore those bytes instead of simply showing "replacement character" as a naive implementation might.

ASGI is decoding headers with latin-1 https://github.com/sanic-org/sanic/blob/ad4e526c775fc3ce950503d6476d9d344492b0dd/sanic/asgi.py#L123
@ahopkins Might wish to have a consistent implementation here?
Yes. @ChihweiLHBird expressed interest in taking this one on.
> ASGI is decoding headers with latin-1

I am a little confused about this... If it is encoded by `utf-8`, which is the general way, can it still be decoded by `latin-1`? Is `latin-1` a superset of `utf8`? Or are we going to also decode header as `utf-8` in asgi?


Resolving patch:
diff --git a/sanic/asgi.py b/sanic/asgi.py
--- a/sanic/asgi.py
+++ b/sanic/asgi.py
@@ -6,7 +6,7 @@
 from urllib.parse import quote
 
 from sanic.compat import Header
-from sanic.exceptions import ServerError
+from sanic.exceptions import BadRequest, ServerError
 from sanic.helpers import Default
 from sanic.http import Stage
 from sanic.log import error_logger, logger
@@ -132,12 +132,20 @@ async def create(
         instance.sanic_app.state.is_started = True
         setattr(instance.transport, "add_task", sanic_app.loop.create_task)
 
-        headers = Header(
-            [
-                (key.decode("latin-1"), value.decode("latin-1"))
-                for key, value in scope.get("headers", [])
-            ]
-        )
+        try:
+            headers = Header(
+                [
+                    (
+                        key.decode("ASCII"),
+                        value.decode(errors="surrogateescape"),
+                    )
+                    for key, value in scope.get("headers", [])
+                ]
+            )
+        except UnicodeDecodeError:
+            raise BadRequest(
+                "Header names can only contain US-ASCII characters"
+            )
         path = (
             scope["path"][1:]
             if scope["path"].startswith("/")


Test patch:
diff --git a/tests/test_asgi.py b/tests/test_asgi.py
--- a/tests/test_asgi.py
+++ b/tests/test_asgi.py
@@ -7,6 +7,9 @@
 import pytest
 import uvicorn
 
+from httpx import Headers
+from pytest import MonkeyPatch
+
 from sanic import Sanic
 from sanic.application.state import Mode
 from sanic.asgi import ASGIApp, Lifespan, MockTransport
@@ -626,3 +629,26 @@ async def before_server_stop(_):
             )
         ]
     )
+
+
+@pytest.mark.asyncio
+async def test_asgi_headers_decoding(app: Sanic, monkeypatch: MonkeyPatch):
+    @app.get("/")
+    def handler(request: Request):
+        return text("")
+
+    headers_init = Headers.__init__
+
+    def mocked_headers_init(self, *args, **kwargs):
+        if "encoding" in kwargs:
+            kwargs.pop("encoding")
+        headers_init(self, encoding="utf-8", *args, **kwargs)
+
+    monkeypatch.setattr(Headers, "__init__", mocked_headers_init)
+
+    message = "Header names can only contain US-ASCII characters"
+    with pytest.raises(BadRequest, match=message):
+        _, response = await app.asgi_client.get("/", headers={"😂": "😅"})
+
+    _, response = await app.asgi_client.get("/", headers={"Test-Header": "😅"})
+    assert response.status_code == 200