Repository: encode/uvicorn

Issue:
WebSocket headers encoding problem
### Checklist

<!-- Please make sure you check all these items before submitting your bug report. -->

- [x] The bug is reproducible against the latest release and/or `master`.
- [x] There are no similar issues or pull requests to fix it yet.

### Describe the bug

<!-- A clear and concise description of what the bug is. -->
If there is a websocket connection with a non-ascii char in header, the server will explode. The problem seems to be from here:
https://github.com/encode/uvicorn/blob/9dc5a43209fe081ba3e74135189252bfddf75587/uvicorn/protocols/websockets/websockets_impl.py#L102
Also, I checked the [ASGI doc](https://buildmedia.readthedocs.org/media/pdf/asgi/latest/asgi.pdf) and in section 2.2.5 it states:
- These are byte strings of the exact byte sequences sent by the client/to be sent by the server. While
modern HTTP standards say that headers should be ASCII, older ones did not and allowed a wider range of
characters. Frameworks/applications should decode headers as they deem appropriate.

So, I think uvicorn should allow for non-ascii chars in header. It is definitely the only reason that we can't use uvicorn in production yet.


### To reproduce

<!-- Provide a *minimal* example with steps to reproduce the bug locally.
Run a websocket server. Then do a ws request with non-unicode header to it.
```bash
websocat -H "X-Test: Abraão" ws://localhost:5000/ws
```

NOTE: try to keep any external dependencies *at an absolute minimum* .
In other words, remove anything that doesn't make the bug go away.

-->

### Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
Doesn't explode.

### Actual behavior

<!-- A clear and concise description of what actually happens. -->
Explodes with an exception.

### Debugging material

<!-- Any tracebacks, screenshots, etc. that can help understanding the problem.

NOTE:
- Please list tracebacks in full (don't truncate them).
- If relevant, consider turning on DEBUG or TRACE logs for additional details (see the Logging section on https://www.uvicorn.org/settings/ specifically the `log-level` flag).
- Consider using `<details>` to make tracebacks/logs collapsible if they're very large (see https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d).
-->

### Environment

- OS / Python / Uvicorn version: just run `uvicorn --version`
- The exact command you're running uvicorn with, all flags you passed included. If you run it with gunicorn please do the same. If there is a reverse-proxy involved and you cannot reproduce without it please give the minimal config of it to reproduce.

### Additional context

<!-- Any additional information that can help understanding the problem.

Eg. linked issues, or a description of what you were trying to achieve. -->


Discussion:
rfc specifically states non-ascii is forbidden https://tools.ietf.org/html/rfc7230#section-1.2

not sure what we should do here, briefly looked at hypercorn and they seem to also use ascii.
curious what others think about this
A lot of tools and and frameworks put arbitrary data in the headers. For example in our instance our 2FA service provider puts the username in the header. And you can guess what kind of characters can appear in the usernames. uvicorn not being able to handle that means that we can not use it in real world scenarios. I'm not saying uvicorn should stop having ascii as the default charset there. I'm saying it should at least __not explode__. Maybe catch the exception and do something else then? Or there would be an option or a hook or some other way to customize it at least.
Imagine that you want to deploy your application behind a proxy or a load balancer that doesn't respect ASCII encoding. Would you say that it's OK for uvicorn to not be able to handle it?
If you are OK with it, we could try to come up with some PR. But I need a suggestion from the team which kind of solution would you prefer.
just played 5min with this, it seems this is only an issue with the websockets implementation, I could send a header with non ascii just fine with wsproto, which is surprising as they tend to be more strict, so you might use that meanwhile.

when you say explode, you mean getting a 500 right ?
encoding this to utf-8 should be ok, would love inputs from others as I'm not sure about unintended consequences of this, any idea @florimondmanca ?
@euri10 This is the same problem as non-conformant HTTP headers. I'd suggest taking a look at what tactics we adopt in HTTPX. Essentially anything non-ascii is off-spec, and you're guaranteed any particular behaviours. Possible tactics include:

* Strictly decode as ascii, loud fail on anything else.
* Strictly decode as some single-byte encoding that's a superset of ascii.
* Decode as UTF-8 first, then fallback to a single-byte encoding that's a superset of ascii.

I think we use the last one of those in HTTPX?

Resolving patch:
diff --git a/uvicorn/protocols/websockets/websockets_impl.py b/uvicorn/protocols/websockets/websockets_impl.py
--- a/uvicorn/protocols/websockets/websockets_impl.py
+++ b/uvicorn/protocols/websockets/websockets_impl.py
@@ -99,7 +99,7 @@ async def process_request(self, path, headers):
             subprotocols.extend([token.strip() for token in header.split(",")])
 
         asgi_headers = [
-            (name.encode("ascii"), value.encode("ascii"))
+            (name.encode("ascii"), value.encode("ascii", errors="surrogateescape"))
             for name, value in headers.raw_items()
         ]
 


Test patch:
diff --git a/tests/protocols/test_websocket.py b/tests/protocols/test_websocket.py
--- a/tests/protocols/test_websocket.py
+++ b/tests/protocols/test_websocket.py
@@ -139,10 +139,13 @@ async def websocket_connect(self, message):
             headers = self.scope.get("headers")
             headers = dict(headers)
             assert headers[b"host"].startswith(b"127.0.0.1")
+            assert headers[b"username"] == bytes("abraão", "utf-8")
             await self.send({"type": "websocket.accept"})
 
     async def open_connection(url):
-        async with websockets.connect(url) as websocket:
+        async with websockets.connect(
+            url, extra_headers=[("username", "abraão")]
+        ) as websocket:
             return websocket.open
 
     config = Config(app=App, ws=protocol_cls, lifespan="off")