Repository: aio-libs/aiohttp

Issue:
Response connection is None on 3.9.0
### Describe the bug

Sorry for the not easily reproducible bug, but since 3.9.0 connection upgrading doesn't work


### To Reproduce

I experienced the bug using aiodocker exec, so forgive me for indirect reproduce of the bug:
* get a running docker container in aiodocker
* `exec_ = await container.exec(command)` (the command doesn't matter, can be ls)
* `async with exec_.start(detach=False) as stream:`
* the above line will fail

### Expected behavior

I used aiodocker version 0.21.0, when using aiohttp version 3.8.6 it works well (but uninstallable for python3.12), on version 3.9.0 it fails with  Status code: 101 (https://github.com/aio-libs/aiodocker/issues/832)

### Logs/tracebacks

```python-traceback
Unfortunately, I only got aiodocker logs:

aiodocker.exceptions.DockerError: DockerError(500, "Cannot upgrade connection to vendored tcp protocol, the docker server has closed underlying socket. Status code: 101. Headers: <CIMultiDictProxy('Content-Type': 'application/vnd.docker.multiplexed-stream', 'Connection': 'Upgrade', 'Upgrade': 'tcp', 'Api-Version': '1.43', 'Docker-Experimental': 'false', 'Ostype': 'linux', 'Server': 'Docker/24.0.7 (linux)')>.")
```
```


### Python Version

```console
$ python --version
Tested on 3.12 and 3.11
```


### aiohttp Version

```console
$ python -m pip show aiohttp
works on 3.8.6, fails on 3.9.0
```


### multidict Version

```console
$ python -m pip show multidict
6.0.4
```


### yarl Version

```console
$ python -m pip show yarl
1.9.3
```


### OS

Ubuntu

### Related component

Client

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct


Discussion:
Unfortunately, thats not a lot to go on.

Any chance that you can bisect to find the regression?

I fail building it with:
`cc1: fatal error: aiohttp/_websocket.c: No such file or directory`
Is there a building documentation?
[#79f5266518b58e1a778450c9b03bf337a7e01901](https://github.com/aio-libs/aiohttp/commit/79f5266518b58e1a778450c9b03bf337a7e01901) is the first bad commit
So it would seem that it’s likely that fixing https://github.com/aio-libs/aiohttp/issues/7864 will also fix this problem as well.  The good news is that one has a reproducer so it will be a bit easier to track down.
Just noticed you linked a PR. Will try that with https://github.com/aio-libs/aiohttp/issues/7864 
Looks like https://github.com/aio-libs/aiohttp/issues/7864 still fails with that change so something more is going on
Looks like https://github.com/aio-libs/aiohttp/issues/7864 still fails with that change so something more is going on
My problem is not related to #7864 
I tried the current master (after #7869) and the issue still exists.
Removing `await self._wait_released()` from read solves the issue on the current master also.
Why does `read` need to release the connection?
Finally was able to reproduce without aiodocker (by copying the interesting parts).
First you need to create a docker container (I used the official Ubuntu one) and replace `container_id` with your container id.
```python
import json

import aiohttp
import asyncio

async def main():
    container_id = 'abaf180acb95e32cc60ed699a99b5756ef8a48c22a7d92cffdf98d30ef7b59f2'
    async with aiohttp.ClientSession(connector=aiohttp.UnixConnector('/run/docker.sock')) as session:
        response = await session.request(
            'POST', 
            f'unix://localhost/v1.43/containers/{container_id}/exec',
            headers={'Content-Type': 'application/json'},
            data=json.dumps({
                'Container': container_id, 
                'Privileged': False, 'Tty': False, 'AttachStdin': False, 'AttachStdout': True, 
                'AttachStderr': True,  'Cmd': ['ls'], 'Env': None, 'WorkingDir': '', 'detachKeys': '', 'User': ''
            })
        )
        exec_id = (await response.json())['Id']
        response = await session.request(
            'POST', 
            f'unix://localhost/v1.43/exec/{exec_id}/start',
            headers={'Content-Type': 'application/json', 'Connection': 'Upgrade', 'Upgrade': 'tcp'},
            data=json.dumps({'Detach': False, 'Tty': False}),
            read_until_eof=False
        )
        data = await response.read()
        conn = response.connection
        print(conn)

asyncio.run(main())
```

The `print(conn)` should not print `None` but `Connection<ConnectionKey(host='localhost', port=None, is_ssl=False, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)>`
I might be wrong, but I just realised 3.9.0 might have broken the all `Connection: Upgrade` header support
> Why does `read` need to release the connection?

I believe it is documented behaviour that has always existed. There is no reason not to release the connection if we have finished reading from it.

The change in 3.9 is more subtle, it now waits for the writer to complete before releasing the connection. It's possible a subtle change in that logic was overlooked and causing some issue.
> I might be wrong, but I just realised 3.9.0 might have broken the all `Connection: Upgrade` header support

So is that mean upgrading a connection is no longer available (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade)?
Shouldn't be, there are several tests for upgrade which are still passing.
> Finally was able to reproduce without aiodocker (by copying the interesting parts). First you need to create a docker container (I used the official Ubuntu one) and replace `container_id` with your container id.
> 
> ```python
> import json
> 
> import aiohttp
> import asyncio
> 
> async def main():
>     container_id = 'abaf180acb95e32cc60ed699a99b5756ef8a48c22a7d92cffdf98d30ef7b59f2'
>     async with aiohttp.ClientSession(connector=aiohttp.UnixConnector('/run/docker.sock')) as session:
>         response = await session.request(
>             'POST', 
>             f'unix://localhost/v1.43/containers/{container_id}/exec',
>             headers={'Content-Type': 'application/json'},
>             data=json.dumps({
>                 'Container': container_id, 
>                 'Privileged': False, 'Tty': False, 'AttachStdin': False, 'AttachStdout': True, 
>                 'AttachStderr': True,  'Cmd': ['ls'], 'Env': None, 'WorkingDir': '', 'detachKeys': '', 'User': ''
>             })
>         )
>         exec_id = (await response.json())['Id']
>         response = await session.request(
>             'POST', 
>             f'unix://localhost/v1.43/exec/{exec_id}/start',
>             headers={'Content-Type': 'application/json', 'Connection': 'Upgrade', 'Upgrade': 'tcp'},
>             data=json.dumps({'Detach': False, 'Tty': False}),
>             read_until_eof=False
>         )
>         data = await response.read()
>         conn = response.connection
>         print(conn)
> 
> asyncio.run(main())
> ```
> 
> The `print(conn)` should not print `None` but `Connection<ConnectionKey(host='localhost', port=None, is_ssl=False, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)>`

So I guess I am using the API incorrectly (and so the author of aiodocker), can you please help me with understanding what I am missing?
There may still be a regression, it's just probably a more subtle edge case than you thought. It'd go a long way if you can add a test in a PR which reproduces the issue.
Will a test using docker (same as the code above) will be ok? Can you please refer me to the current tests?
Probably not, we don't have docker in the CI, and we have downstream packagers who need to run tests without docker.

Test would probably go in
https://github.com/aio-libs/aiohttp/blob/master/tests/test_client_request.py
or
https://github.com/aio-libs/aiohttp/blob/master/tests/test_client_functional.py
I tried writing a test. but since I am not familiar with your code it might take me a couple of weeks, your help would really be appreciated (a release without breaking the API but with python3.12 support would also be appreciated)
> The `print(conn)` should not print `None` but `Connection<ConnectionKey(host='localhost', port=None, is_ssl=False, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)>`

Wait, I missed this. This should print `None`, because the connection is closed and removed. Why do you think it shouldn't be? This should happen on older releases too.
In 3.8.6 it keeps the connection and aiodocker uses this to upgrade the connection to tcp
> In 3.8.6 it keeps the connection and aiodocker uses this to upgrade the connection to tcp

We have tests on 3.8 that say it shouldn't:
https://github.com/aio-libs/aiohttp/blob/b51610b93b2ae15c4062e3a1680a536ba5f4c5c4/tests/test_client_response.py#L220-L222

I think what you want to be printing is `client.connector._conns`. There should be a connection open there after the read().
Wait a moment, I didn't see `read_until_eof=False`. Maybe this is the problem. Let me have a look at how that's supposed to work...
It seems to me that if you use that parameter, then `resp.read()` is always going to return an empty value, if I understand the code correctly. Not really sure how that's useful, but the fix seems simple, we just need to skip the wait_released when that setting is False.
From a quick look I saw that response is not accessible to the `read_until_eof` setting, do you prefer adding it yourself or shall I add it to response class and open a PR?
I'm just sorting out a test at the moment. I think that attribute is accessible, but will get the test sorted first.
Hmm, struggling to get a test that validates. Do you know what the server response looks like?

This fails on 3.8:
```
async def test_no_eof_response_not_released(aiohttp_client) -> None:
    async def handler(request):
        body = await request.read()
        assert b"" == body
        return web.Response(body=b"OK" * 2000)

    app = web.Application()
    app.router.add_route("GET", "/", handler)

    client = await aiohttp_client(app)

    resp = await client.get("/")
    await resp.read()
    assert resp.connection is None
    resp = await client.get("/", read_until_eof=False)
    await resp.read()
    assert resp.connection is not None
```
I am not sure, but when printing `raw_headers` and the data I get:
```
((b'Content-Type', b'application/vnd.docker.multiplexed-stream'), (b'Connection', b'Upgrade'), (b'Upgrade', b'tcp'), (b'Api-Version', b'1.43'), (b'Docker-Experimental', b'false'), (b'Ostype', b'linux'), (b'Server', b'Docker/24.0.7 (linux)'))
b''
```
Hmm, I can only get it to fail or to hang...
>  Do you know what the server response looks like?


@matan1008 
You can run the code with

`strace -f -s 4096 -o calls.log <your script>`

Than you can extract the raw read/write calls from the calls.log file to see whats actually being sent

This is the request:
```
POST /v1.43/exec/206c8c5da12eda3c5e9b68435dd81bd3aa8a010cd73ed1305e6a3b090d2deba5/start HTTP/1.1
Host: localhost
Content-Type: application/json
Connection: Upgrade
Upgrade: tcp
Accept: */*
Accept-Encoding: gzip, deflate
User-Agent: Python/3.11 aiohttp/3.8.6
Content-Length: 31
```

And this is the response:
```
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.multiplexed-stream
Connection: Upgrade
Upgrade: tcp
Api-Version: 1.43
Docker-Experimental: false
Ostype: linux
Server: Docker/24.0.7 (linux)
```
OK, it's completely unrelated to that parameter (which I still don't understand what it does). Should have a fix shortly.
Nope, I take it back. This test reproduces without extensions:

```
async def test_upgrade_connection_not_released_after_read(aiohttp_client: Any) -> None:
    async def handler(request: web.Request) -> web.Response:
        body = await request.read()
        assert b"" == body
        return web.Response(status=101, headers={"Connection": "Upgrade"})

    app = web.Application()
    app.router.add_route("GET", "/", handler)

    client = await aiohttp_client(app)

    resp = await client.get("/")
    await resp.read()
    assert resp.connection is not None
```

But, still fails with extensions. Is it possible you've installed a version without wheels, or otherwise disabled the extensions?
(You can check at runtime by checking what `aiohttp.http_parser.HttpResponseParser` is, by printing it or whatever. The compiled one is `aiohttp._http_parser.HttpResponseParser` (with _), the Python one is `aiohttp.http_parser.HttpResponseParser`).
I installed it regularly with pip (through aiodocker)
Ah, I think I was right previously. It seems that the C parser only considers the connection upgraded when the Upgrade header is present, whereas the Py parser doesn't care if that header is present. Probably a bug to fix in the Py parser there.

If you can test the change in the PR to double check it solves the problem for you, that'd be great.
Will do tomorrow morning, thanks!
Works great, Thank you!

Resolving patch:
diff --git a/aiohttp/client_reqrep.py b/aiohttp/client_reqrep.py
--- a/aiohttp/client_reqrep.py
+++ b/aiohttp/client_reqrep.py
@@ -1006,19 +1006,14 @@ def _response_eof(self) -> None:
         if self._closed:
             return
 
-        if self._connection is not None:
-            # websocket, protocol could be None because
-            # connection could be detached
-            if (
-                self._connection.protocol is not None
-                and self._connection.protocol.upgraded
-            ):
-                return
-
-            self._release_connection()
+        # protocol could be None because connection could be detached
+        protocol = self._connection and self._connection.protocol
+        if protocol is not None and protocol.upgraded:
+            return
 
         self._closed = True
         self._cleanup_writer()
+        self._release_connection()
 
     @property
     def closed(self) -> bool:
@@ -1113,7 +1108,9 @@ async def read(self) -> bytes:
         elif self._released:  # Response explicitly released
             raise ClientConnectionError("Connection closed")
 
-        await self._wait_released()  # Underlying connection released
+        protocol = self._connection and self._connection.protocol
+        if protocol is None or not protocol.upgraded:
+            await self._wait_released()  # Underlying connection released
         return self._body  # type: ignore[no-any-return]
 
     def get_encoding(self) -> str:
diff --git a/aiohttp/connector.py b/aiohttp/connector.py
--- a/aiohttp/connector.py
+++ b/aiohttp/connector.py
@@ -127,6 +127,10 @@ def __del__(self, _warnings: Any = warnings) -> None:
                 context["source_traceback"] = self._source_traceback
             self._loop.call_exception_handler(context)
 
+    def __bool__(self) -> Literal[True]:
+        """Force subclasses to not be falsy, to make checks simpler."""
+        return True
+
     @property
     def loop(self) -> asyncio.AbstractEventLoop:
         warnings.warn(


Test patch:
diff --git a/tests/test_client_functional.py b/tests/test_client_functional.py
--- a/tests/test_client_functional.py
+++ b/tests/test_client_functional.py
@@ -173,6 +173,25 @@ async def handler(request):
     assert 1 == len(client._session.connector._conns)
 
 
+async def test_upgrade_connection_not_released_after_read(aiohttp_client) -> None:
+    async def handler(request: web.Request) -> web.Response:
+        body = await request.read()
+        assert b"" == body
+        return web.Response(
+            status=101, headers={"Connection": "Upgrade", "Upgrade": "tcp"}
+        )
+
+    app = web.Application()
+    app.router.add_route("GET", "/", handler)
+
+    client = await aiohttp_client(app)
+
+    resp = await client.get("/")
+    await resp.read()
+    assert resp.connection is not None
+    assert not resp.closed
+
+
 async def test_keepalive_server_force_close_connection(aiohttp_client) -> None:
     async def handler(request):
         body = await request.read()