Repository: encode/httpx

Issue:
HTTP/2 connection implementation is currently flaky.
Is it me ?
Or the simplest test fails:

     httpcore.Client().get("https://www.google.com")

outputs:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/dist-packages/httpcore/concurrency.py", line 65, in read
        self.stream_reader.read(n), timeout.read_timeout
      File "/usr/lib/python3.7/asyncio/tasks.py", line 423, in wait_for
        raise futures.TimeoutError()
    concurrent.futures._base.TimeoutError

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/connection_pool.py", line 114, in send
        request, stream=stream, ssl=ssl, timeout=timeout
      File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/connection.py", line 53, in send
        request, stream=stream, timeout=timeout
      File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/http2.py", line 77, in send
        await response.read()
      File "/usr/local/lib/python3.7/dist-packages/httpcore/models.py", line 684, in read
        self._content = b"".join([part async for part in self.stream()])
      File "/usr/local/lib/python3.7/dist-packages/httpcore/models.py", line 684, in <listcomp>
        self._content = b"".join([part async for part in self.stream()])
      File "/usr/local/lib/python3.7/dist-packages/httpcore/models.py", line 695, in stream
        async for chunk in self.raw():
      File "/usr/local/lib/python3.7/dist-packages/httpcore/models.py", line 712, in raw
        async for part in self._raw_stream:
      File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/http2.py", line 121, in body_iter
        event = await self.receive_event(stream_id, timeout)
      File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/http2.py", line 131, in receive_event
        data = await self.reader.read(self.READ_NUM_BYTES, timeout)
      File "/usr/local/lib/python3.7/dist-packages/httpcore/concurrency.py", line 68, in read
        raise ReadTimeout()
    httpcore.exceptions.ReadTimeout

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/manatlan/Bureau/reqman/reqman.py", line 1414, in <module>
        httpcore.Client().get("https://www.google.com")
      File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 496, in get
        timeout=timeout,
      File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 472, in request
        timeout=timeout,
      File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 665, in send
        timeout=timeout,
      File "/usr/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
        return future.result()
      File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 287, in send
        allow_redirects=allow_redirects,
      File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 313, in send_handling_redirects
        request, stream=stream, ssl=ssl, timeout=timeout
      File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/connection_pool.py", line 117, in send
        self.active_connections.remove(connection)
      File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/connection_pool.py", line 64, in remove
        del self.all[connection]
    KeyError: <httpcore.dispatch.connection.HTTPConnection object at 0x7fadbf88e0f0>

btw, I can't catch the Exception ?!

    try:
        httpcore.Client().get("https://www.google.com")
    except concurrent.futures._base.TimeoutError:
        print(1)
    except httpcore.exceptions.ReadTimeout:
        print(2)


Discussion:
Thanks for the report. TL;DR Google is using HTTP2 and returning a 400 error which creates an unexpected content which causes the ReadTimeout. I've tried with other URLs and it seems to work correctly.

Looking deeper there are few things happening:

- Firstly Google is returning a 400 response and resetting the stream, here are the raw events:
```
  <ResponseReceived stream_id:1, headers:[(b':status', b'400'), (b'content-type', b'text/html; charset=UTF-8'), (b'referrer-policy', b'no-referrer'), (b'content-length', b'1555'), (b'date', b'Sat, 25 May 2019 10:25:58 GMT')]>
  <StreamReset stream_id:1, error_code:ErrorCodes.PROTOCOL_ERROR, remote_reset:True>
```
- The HTTP2 dispatcher attempts to get a body iterator for the content on `content = self.body_iter(stream_id)`, which is then passed to the response.
- Further on `await response.read()` the `body_iter` attempts to read events from the stream and yield data, however the `StreamReset` is the last event of the stream which is unhandled in the loop, so it attempts to read a new event from the stream and times out.
- The `ReadTimeout` exception is caught in the `try..except` and closes the response in the finally block, the `ReadTimeout` bubbles up
- The `ConnectionPool.send` method also has a `try..except` block which catches the timeout exception and attempts to remove the connection there, but the connection has already been removed on `response.close` so a `KeyError` is raised.

I'll see if I can put together a couple of PRs for these issues.
Thanks folks. I should be able to review this on Tuesday.
Something that’d be really useful for us with debugging would be configuration on the client to determine which HTTP protocols should be supported. Then if we get something like this we can easily narrow it down to eg. “Is this an issue in the HTTP/2 implementation”
It's specifically an issue with HTTP/2 (doesn't reproduce if I switch the connection to force HTTP/1). Doesn't reproduce on all sites (eg. I'd be testing against the HTTP/2 implementation against `https://www.example.com`.)
thanks in advance ...
I will use httpcore in https://github.com/manatlan/reqman ...

Hope you release a httpcore on pypi with cookies soon

Resolving patch:
diff --git a/httpcore/dispatch/http11.py b/httpcore/dispatch/http11.py
--- a/httpcore/dispatch/http11.py
+++ b/httpcore/dispatch/http11.py
@@ -46,6 +46,9 @@ async def send(
         method = request.method.encode("ascii")
         target = request.url.full_path.encode("ascii")
         headers = request.headers.raw
+        if 'Host' not in request.headers:
+            host = request.url.authority.encode("ascii")
+            headers = [(b"host", host)] + headers
         event = h11.Request(method=method, target=target, headers=headers)
         await self._send_event(event, timeout)
 
diff --git a/httpcore/models.py b/httpcore/models.py
--- a/httpcore/models.py
+++ b/httpcore/models.py
@@ -523,14 +523,11 @@ async def stream(self) -> typing.AsyncIterator[bytes]:
     def prepare(self) -> None:
         auto_headers = []  # type: typing.List[typing.Tuple[bytes, bytes]]
 
-        has_host = "host" in self.headers
         has_content_length = (
             "content-length" in self.headers or "transfer-encoding" in self.headers
         )
         has_accept_encoding = "accept-encoding" in self.headers
 
-        if not has_host and self.url.authority:
-            auto_headers.append((b"host", self.url.authority.encode("ascii")))
         if not has_content_length:
             if self.is_streaming:
                 auto_headers.append((b"transfer-encoding", b"chunked"))


Test patch:
diff --git a/tests/models/test_requests.py b/tests/models/test_requests.py
--- a/tests/models/test_requests.py
+++ b/tests/models/test_requests.py
@@ -8,11 +8,11 @@ def test_request_repr():
     assert repr(request) == "<Request('GET', 'http://example.org')>"
 
 
-def test_host_header():
+def test_no_content():
     request = httpcore.Request("GET", "http://example.org")
     request.prepare()
     assert request.headers == httpcore.Headers(
-        [(b"host", b"example.org"), (b"accept-encoding", b"deflate, gzip, br")]
+        [(b"accept-encoding", b"deflate, gzip, br")]
     )
 
 
@@ -21,7 +21,6 @@ def test_content_length_header():
     request.prepare()
     assert request.headers == httpcore.Headers(
         [
-            (b"host", b"example.org"),
             (b"content-length", b"8"),
             (b"accept-encoding", b"deflate, gzip, br"),
         ]
@@ -33,7 +32,6 @@ def test_url_encoded_data():
     request.prepare()
     assert request.headers == httpcore.Headers(
         [
-            (b"host", b"example.org"),
             (b"content-length", b"8"),
             (b"accept-encoding", b"deflate, gzip, br"),
             (b"content-type", b"application/x-www-form-urlencoded"),
@@ -52,7 +50,6 @@ async def streaming_body(data):
     request.prepare()
     assert request.headers == httpcore.Headers(
         [
-            (b"host", b"example.org"),
             (b"transfer-encoding", b"chunked"),
             (b"accept-encoding", b"deflate, gzip, br"),
         ]
@@ -75,7 +72,7 @@ def test_override_accept_encoding_header():
     request = httpcore.Request("GET", "http://example.org", headers=headers)
     request.prepare()
     assert request.headers == httpcore.Headers(
-        [(b"host", b"example.org"), (b"accept-encoding", b"identity")]
+        [(b"accept-encoding", b"identity")]
     )
 
 
@@ -90,7 +87,6 @@ async def streaming_body(data):
     request.prepare()
     assert request.headers == httpcore.Headers(
         [
-            (b"host", b"example.org"),
             (b"accept-encoding", b"deflate, gzip, br"),
             (b"content-length", b"8"),
         ]