Repository: aio-libs/aiohttp

Issue:
Server using Python parser responds 500 for invalid Unicode in header
### Describe the bug

When an AIOHTTP server receives a request containing an invalid header, it formulates a response that echos the invalid header back to the user.

Something like this:
```
printf 'GET / HTTP/1.1\r\nI am invalid!!\r\n\r\n' | nc localhost 8080
```
gets a response like this:
```
HTTP/1.0 400 Bad Request
Content-Type: text/plain; charset=utf-8
Content-Length: 35
Date: Mon, 16 Oct 2023 19:27:30 GMT
Server: Python/3.11 aiohttp/4.0.0a2.dev0

Invalid HTTP Header: I am invalid!!
```

If the invalid header contains dangling UTF-8 surrogates, then the server is unable to encode the received bytes into Unicode, so the default error handler fails, and the server instead responds 500.

For example, something like this:
```
printf 'GET / HTTP/1.1\r\n\xff\r\n\r\n' | nc localhost 8080
```
gets a response like this:
```
HTTP/1.0 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
Content-Length: 55
Date: Mon, 16 Oct 2023 19:26:29 GMT
Server: Python/3.11 aiohttp/4.0.0a2.dev0

500 Internal Server Error

Server got itself in trouble
```

### To Reproduce

0. Install aiohttp
1. Start the example server:
```
export AIOHTTP_NO_EXTENSIONS=1
python3 examples/server_simple.py
```
2. Send it a request:
```
printf 'GET / HTTP/1.1\r\n\xff\r\n\r\n' | nc localhost 8080
```
3. Observe that it responds 500:
```
HTTP/1.0 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
Content-Length: 55
Date: Mon, 16 Oct 2023 19:33:03 GMT
Server: Python/3.11 aiohttp/4.0.0a2.dev0

500 Internal Server Error

Server got itself in trouble
```

### Expected behavior

The server should have responded 400.

### Logs/tracebacks

```python-traceback
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
Error handling request
Traceback (most recent call last):
  File "/home/bkallus/clones/aiohttp/aiohttp/web_protocol.py", line 366, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bkallus/clones/aiohttp/aiohttp/http_parser.py", line 314, in feed_data
    msg: _MsgT = self.parse_message(self._lines)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bkallus/clones/aiohttp/aiohttp/http_parser.py", line 601, in parse_message
    ) = self.parse_headers(lines)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bkallus/clones/aiohttp/aiohttp/http_parser.py", line 466, in parse_headers
    headers, raw_headers = self._headers_parser.parse_headers(lines)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bkallus/clones/aiohttp/aiohttp/http_parser.py", line 139, in parse_headers
    raise InvalidHeader(line) from None
aiohttp.http_exceptions.InvalidHeader: 400, message:
  Invalid HTTP Header: \udcff
Error handling request
Traceback (most recent call last):
  File "/home/bkallus/clones/aiohttp/aiohttp/web_protocol.py", line 468, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bkallus/clones/aiohttp/aiohttp/web_protocol.py", line 707, in handler
    return self.handle_error(
           ^^^^^^^^^^^^^^^^^^
  File "/home/bkallus/clones/aiohttp/aiohttp/web_protocol.py", line 698, in handle_error
    resp = Response(status=status, text=message, content_type=ct)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bkallus/clones/aiohttp/aiohttp/web_response.py", line 539, in __init__
    body = text.encode(charset)
           ^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 21: surrogates not allowed
```


### Python Version

```console
$ python --version
Python 3.11.5
```


### aiohttp Version

```console
$ python -m pip show aiohttp
Name: aiohttp
Version: 4.0.0a2.dev0
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Location: /home/bkallus/clones/aiohttp/env/lib/python3.11/site-packages
Editable project location: /home/bkallus/clones/aiohttp
Requires: aiosignal, frozenlist, multidict, yarl
Required-by:
```


### multidict Version

```console
$ python -m pip show multidict
Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /home/bkallus/clones/aiohttp/env/lib/python3.11/site-packages
Requires:
Required-by: aiohttp, yarl
```


### yarl Version

```console
$ python -m pip show yarl
Name: yarl
Version: 1.9.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: /home/bkallus/clones/aiohttp/env/lib/python3.11/site-packages
Requires: idna, multidict
Required-by: aiohttp
```


### OS

Arch Linux (Linux 6.1.56-1-lts)

### Related component

Server

### Additional context

This bug (as with the others I've found) was discovered by differential fuzzing against a Node-based HTTP server.

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct


Discussion:

Resolving patch:
diff --git a/aiohttp/http_exceptions.py b/aiohttp/http_exceptions.py
--- a/aiohttp/http_exceptions.py
+++ b/aiohttp/http_exceptions.py
@@ -87,10 +87,9 @@ def __init__(
 
 class InvalidHeader(BadHttpMessage):
     def __init__(self, hdr: Union[bytes, str]) -> None:
-        if isinstance(hdr, bytes):
-            hdr = hdr.decode("utf-8", "surrogateescape")
-        super().__init__(f"Invalid HTTP Header: {hdr}")
-        self.hdr = hdr
+        hdr_s = hdr.decode(errors="backslashreplace") if isinstance(hdr, bytes) else hdr
+        super().__init__(f"Invalid HTTP header: {hdr!r}")
+        self.hdr = hdr_s
         self.args = (hdr,)
 
 
diff --git a/aiohttp/http_parser.py b/aiohttp/http_parser.py
--- a/aiohttp/http_parser.py
+++ b/aiohttp/http_parser.py
@@ -595,7 +595,9 @@ def parse_message(self, lines: List[bytes]) -> RawRequestMessage:
             url = URL(path, encoded=True)
             if url.scheme == "":
                 # not absolute-form
-                raise InvalidURLError(line)
+                raise InvalidURLError(
+                    path.encode(errors="surrogateescape").decode("latin1")
+                )
 
         # read headers
         (


Test patch:
diff --git a/tests/test_http_exceptions.py b/tests/test_http_exceptions.py
--- a/tests/test_http_exceptions.py
+++ b/tests/test_http_exceptions.py
@@ -102,7 +102,7 @@ class TestInvalidHeader:
     def test_ctor(self) -> None:
         err = http_exceptions.InvalidHeader("X-Spam")
         assert err.code == 400
-        assert err.message == "Invalid HTTP Header: X-Spam"
+        assert err.message == "Invalid HTTP header: 'X-Spam'"
         assert err.headers is None
 
     def test_pickle(self) -> None:
@@ -112,17 +112,17 @@ def test_pickle(self) -> None:
             pickled = pickle.dumps(err, proto)
             err2 = pickle.loads(pickled)
             assert err2.code == 400
-            assert err2.message == "Invalid HTTP Header: X-Spam"
+            assert err2.message == "Invalid HTTP header: 'X-Spam'"
             assert err2.headers is None
             assert err2.foo == "bar"
 
     def test_str(self) -> None:
         err = http_exceptions.InvalidHeader(hdr="X-Spam")
-        assert str(err) == "400, message:\n  Invalid HTTP Header: X-Spam"
+        assert str(err) == "400, message:\n  Invalid HTTP header: 'X-Spam'"
 
     def test_repr(self) -> None:
         err = http_exceptions.InvalidHeader(hdr="X-Spam")
-        expected = "<InvalidHeader: 400, message='Invalid HTTP Header: X-Spam'>"
+        expected = "<InvalidHeader: 400, message=\"Invalid HTTP header: 'X-Spam'\">"
         assert repr(err) == expected
 
 
diff --git a/tests/test_http_parser.py b/tests/test_http_parser.py
--- a/tests/test_http_parser.py
+++ b/tests/test_http_parser.py
@@ -186,6 +186,23 @@ def test_bad_headers(parser: Any, hdr: str) -> None:
         parser.feed_data(text)
 
 
+def test_unpaired_surrogate_in_header_py(loop: Any, protocol: Any) -> None:
+    parser = HttpRequestParserPy(
+        protocol,
+        loop,
+        2**16,
+        max_line_size=8190,
+        max_field_size=8190,
+    )
+    text = b"POST / HTTP/1.1\r\n\xff\r\n\r\n"
+    message = None
+    try:
+        parser.feed_data(text)
+    except http_exceptions.InvalidHeader as e:
+        message = e.message.encode("utf-8")
+    assert message is not None
+
+
 def test_content_length_transfer_encoding(parser: Any) -> None:
     text = (
         b"GET / HTTP/1.1\r\nHost: a\r\nContent-Length: 5\r\nTransfer-Encoding: a\r\n\r\n"
@@ -741,11 +758,16 @@ def test_http_request_parser_bad_version_number(parser: Any) -> None:
         parser.feed_data(b"GET /test HTTP/1.32\r\n\r\n")
 
 
-def test_http_request_parser_bad_uri(parser: Any) -> None:
+def test_http_request_parser_bad_ascii_uri(parser: Any) -> None:
     with pytest.raises(http_exceptions.InvalidURLError):
         parser.feed_data(b"GET ! HTTP/1.1\r\n\r\n")
 
 
+def test_http_request_parser_bad_nonascii_uri(parser: Any) -> None:
+    with pytest.raises(http_exceptions.InvalidURLError):
+        parser.feed_data(b"GET \xff HTTP/1.1\r\n\r\n")
+
+
 @pytest.mark.parametrize("size", [40965, 8191])
 def test_http_request_max_status_line(parser, size) -> None:
     path = b"t" * (size - 5)