Repository: aio-libs/aiohttp

Issue:
Connections are not correctly kept after HEAD requests
When the client makes a HEAD request, it parses the response as if it was a regular message.

But the HTTP specs has a nice trick: HEAD requests `SHOULD` respond with the exact same headers than the corresponding GET request. This includes `Content-Length` or `Transfer-Encoding: chunked` headers.

This means that when the client receives a HEAD response with a `Content-Length: 5` response, it should not expect any body and `payload.is_eof()` should be True immediately, not after receiving 5 more bytes.

Currently `ResponseHandler.should_close` is True for HEAD requests with a `Content-Length: …` because `payload.is_eof()` is False when it should be True, hence the underlying connection is closed and not re-used when it could be.


This is only inefficient on the client side, not at all a bug, and does not produce any wrong results.


(also, I was mistaken a few minutes ago: returning a Content-Length header with an empty body is perfectly valid, and the `allow_head=True` parameter of the `add_get` method works perfectly well, no problem at all on the server side)


reproducer:
==========

https://github.com/aio-libs/aiohttp/issues/1769#issuecomment-614775045


Discussion:
Does aiohttp client support pipelining? If a client sends more than one requests on the same connection before server returns, this may be a problem if the client mistakenly parses the next respond as the respond body.
client does not support pipelining, issue #1740
Dropping in to say that I was bitten by this bug as well, took a few hours to debug why it was happening.
It'd be great if somebody could post a complete reproducer. Or better yet — a test marked with xfail as per https://pganssle-talks.github.io/xfail-lightning.
I have this as a simple reproducer locally FYI:

```python
import aiohttp


async def test_head_request_reuses_connection():
    class SingleTCPConnector(aiohttp.TCPConnector):
        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)
            self._connections_created = 0

        def _create_connection(self, *args, **kwargs):
            self._connections_created += 1
            if self._connections_created > 1:
                raise ValueError("only one connection!")
            return super()._create_connection(*args, **kwargs)

    async with SingleTCPConnector(limit=1) as connector:
        for _ in range(2):
            async with aiohttp.request("HEAD", "http://example.com", connector=connector) as resp:
                await resp.read()
```

Fails with `HEAD` but passes with all other methods.
@sethmlarson yeah, it'd be great if you could send that as a PR.

Resolving patch:
diff --git a/aiohttp/client_proto.py b/aiohttp/client_proto.py
--- a/aiohttp/client_proto.py
+++ b/aiohttp/client_proto.py
@@ -153,6 +153,7 @@ def set_response_params(self, *, timer: BaseTimerContext=None,
         self._parser = HttpResponseParser(
             self, self._loop, timer=timer,
             payload_exception=ClientPayloadError,
+            response_with_body=not skip_payload,
             read_until_eof=read_until_eof,
             auto_decompress=auto_decompress)
 


Test patch:
diff --git a/tests/test_client_functional.py b/tests/test_client_functional.py
--- a/tests/test_client_functional.py
+++ b/tests/test_client_functional.py
@@ -54,6 +54,37 @@ async def handler(request):
     assert 1 == len(client._session.connector._conns)
 
 
+async def test_keepalive_after_head_requests_success(
+        aiohttp_client) -> None:
+    async def handler(request):
+        body = await request.read()
+        assert b'' == body
+        return web.Response(body=b'OK')
+
+    cnt_conn_reuse = 0
+
+    async def on_reuseconn(session, ctx, params):
+        nonlocal cnt_conn_reuse
+        cnt_conn_reuse += 1
+
+    trace_config = aiohttp.TraceConfig()
+    trace_config._on_connection_reuseconn.append(on_reuseconn)
+
+    app = web.Application()
+    app.router.add_route('GET', '/', handler)
+
+    connector = aiohttp.TCPConnector(limit=1)
+    client = await aiohttp_client(app, connector=connector,
+                                  trace_configs=[trace_config])
+
+    resp1 = await client.head('/')
+    await resp1.read()
+    resp2 = await client.get('/')
+    await resp2.read()
+
+    assert 1 == cnt_conn_reuse
+
+
 async def test_keepalive_response_released(aiohttp_client) -> None:
     async def handler(request):
         body = await request.read()