Repository: encode/uvicorn

Issue:
UnicodeDecodeError when decoding bad headers
Someone (or some bot) was spamming my sever with requests to potential vulnerabilities.

One of the attacks is for a potential vulnerability in php, which sets the the `x-forwarded-for` header to the following value:
```
}__test|O:21:"JDatabaseDriverMysqli":3:{s:2:"fc";O:17:"JSimplepieFactory":0:{}s:21:"\\0\\0\\0disconnectHandlers";a:1:{i:0;a:2:{i:0;O:9:"SimplePie":5:{s:8:"sanitize";O:20:"JDatabaseDriverMysql":0:{}s:8:"feed_url";s:56:"die(md5(DIRECTORY_SEPARATOR));JFactory::getConfig();exit";s:19:"cache_name_function";s:6:"assert";s:5:"cache";b:1;s:11:"cache_class";O:20:"JDatabaseDriverMysql":0:{}}i:1;s:4:"init";}}s:13:"\\0\\0\\0connection";b:1;}\xf0\xfd\xfd\xfd, ...
```

This leads to this exception:

```
Exception in ASGI application

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 385, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 40, in __call__
    x_forwarded_for = headers[b"x-forwarded-for"].decode("ascii")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 427: ordinal not in range(128)
```

As it's due to malformed header from the client, I would expect this should be a 400 error instead?


Discussion:
I haven't previously read the module where that error appears, but I've been thinking about headers lately and allowed values, so I was interested to read this issue. 

Reading that module now, I notice there's this docstring:

```
"""
This middleware can be used when a known proxy is fronting the application,
and is trusted to be properly setting the `X-Forwarded-Proto` and
`X-Forwarded-For` headers with the connecting client information.
...
```

It sounds like your application is not behind a proxy? 

Browsers (for one example) will happily send all kinds of junk in headers (which I explored recently when digging into cookies), which makes me wonder what would be the right thing to do with deliberately malicious headers. The [http 1.1 spec](https://www.greenbytes.de/tech/webdav/rfc7230.html#field.components) has some useful suggestions, and those characters at the end would not be valid in a header token, comment, or quote. 

I think I agree that there should be a 400 returned. In any case, I don't think it should throw an exception.
I am behind a proxy (it's running on google cloud kubernetes with a [basic ingress setup](https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress#using-gcloud-init)), but I guess it doesn't fit this part:
> ... and is trusted to be properly setting the `X-Forwarded-Proto` and `X-Forwarded-For` headers ...

the `, ...` part at the end of my header value in the previous post was me removing the real ip addresses. The proxy will chain it's own additions onto the end of any existing `X-Forwarded-For` headers.

e.g. 
if i do 
`curl https://[SERVICE-URL]/` 
I will see 
`X-Forwarded-For: [MY-HOME-IP-ADDRESS], [SERVICE-IP-ADDRESS]`

and if i do 
`curl -H 'X-forwarded-for: some-random-stuff' https://[SERVICE-URL]/`
 I will see 
`X-Forwarded-For: some-random-stuff, [MY-HOME-IP-ADDRESS], [SERVICE-IP-ADDRESS]`

Interestingly, I just realised that i'm always seeing my `[SERVICE-IP-ADDRESS]` in the logs, because the `ProxyHeadersMiddleware` is taking the last value, rather than the first:
```
host = x_forwarded_for.split(",")[-1].strip()
```
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For says
> Syntax
> `X-Forwarded-For: <client>, <proxy1>, <proxy2>`


Resolving patch:
diff --git a/uvicorn/middleware/proxy_headers.py b/uvicorn/middleware/proxy_headers.py
--- a/uvicorn/middleware/proxy_headers.py
+++ b/uvicorn/middleware/proxy_headers.py
@@ -30,14 +30,14 @@ async def __call__(self, scope, receive, send):
                 if b"x-forwarded-proto" in headers:
                     # Determine if the incoming request was http or https based on
                     # the X-Forwarded-Proto header.
-                    x_forwarded_proto = headers[b"x-forwarded-proto"].decode("ascii")
+                    x_forwarded_proto = headers[b"x-forwarded-proto"].decode("latin1")
                     scope["scheme"] = x_forwarded_proto.strip()
 
                 if b"x-forwarded-for" in headers:
                     # Determine the client address from the last trusted IP in the
                     # X-Forwarded-For header. We've lost the connecting client's port
                     # information by now, so only include the host.
-                    x_forwarded_for = headers[b"x-forwarded-for"].decode("ascii")
+                    x_forwarded_for = headers[b"x-forwarded-for"].decode("latin1")
                     host = x_forwarded_for.split(",")[-1].strip()
                     port = 0
                     scope["client"] = (host, port)


Test patch:
diff --git a/tests/middleware/test_proxy_headers.py b/tests/middleware/test_proxy_headers.py
--- a/tests/middleware/test_proxy_headers.py
+++ b/tests/middleware/test_proxy_headers.py
@@ -28,3 +28,14 @@ def test_proxy_headers_no_port():
     response = client.get("/", headers=headers)
     assert response.status_code == 200
     assert response.text == "Remote: https://1.2.3.4:0"
+
+
+def test_proxy_headers_invalid_x_forwarded_for():
+    client = TestClient(app)
+    headers = {
+        "X-Forwarded-Proto": "https",
+        "X-Forwarded-For": "\xf0\xfd\xfd\xfd, 1.2.3.4",
+    }
+    response = client.get("/", headers=headers)
+    assert response.status_code == 200
+    assert response.text == "Remote: https://1.2.3.4:0"