Repository: encode/httpx

Issue:
Unintuitive behavior with using an AsyncClient after it's been aclosed
### Checklist

- [X] The bug is reproducible against the latest release and/or `master`.
- [X] There are no similar issues or pull requests to fix it yet.

### Describe the bug

I'm playing around with httpx to learn how async programming works in Python, and I've noticed it's pretty easy to get into a state where I have an async task using an AsyncClient that keeps running after I've exited the `with` block that created the client, i.e., after the the `__aclose__` method of the client has been called.

Here's a simple example in asyncio:

```python
import httpx
import asyncio

async def get(client, url):
    r = await client.get(url)
    print(r.text)

async def main():
    async with httpx.AsyncClient() as client:
        t1 = asyncio.create_task(get(client, "https://httpbin.org/delay/1"))
        t2 = asyncio.create_task(get(client, "https://httpbin.org/delay/2"))

    await t1
    await t2

if __name__ == "__main__":
    asyncio.run(main())
```

Here's a simple example in trio (where I ran into this):

```python
import httpx
import trio

async def get(client ,url):
    x = await client.get(url)
    print(x.text)

async def main():
    async with trio.open_nursery() as nursery:
        async with httpx.AsyncClient() as client:
            nursery.start_soon(get, client, "https://httpbin.org/delay/1")
            nursery.start_soon(get, client, "https://httpbin.org/delay/2")

if __name__ == "__main__":
    trio.run(main)
```

These produce the same output: they run the two requests in parallel, print the output, and then display the warning
```
.../venv/lib/python3.7/site-packages/httpx/_client.py:1777: UserWarning: Unclosed <httpx.AsyncClient object at 0x7fbe37ad33c8>. See https://www.python-httpx.org/async/#opening-and-closing-clients for details.
  f"Unclosed {self!r}. "
```

It's surprising to me that the code both _works_ and prints the warning. What's actually going on is that the `__aclose__` method sets a flag indicating the client's been closed and then calls `__aclose__` on the transport, which flushes and closes any open connections, but does not actually leave the transport in an unusable state. That means that new connections can still get created. Since in both cases the tasks are scheduled for after the `with` block has exited, I think that they _deterministically_ don't start until after `__aclose__` finishes, ie., they aren't affected by tearing down any existing connections, and they go on to create new ones. Then the `send` method goes and unsets the `_is_closed` flag (which I'm confused about - I see this behavior is intentional per https://github.com/encode/httpx/pull/1197#issuecomment-676489918 but I don't follow why) and you're left with an unclosed client at GC time.

I think it would be helpful for `__aclose__` to leave the client in an unusable state, e.g., set `self._transport = None`, or have `send` fail if `_is_closed` is set, if the intention is that clients should not be reopened once closed. That would prevent users from writing code where they think they're cleaning up properly, but they're actually cleaning up too early, while requests are still in flight.

If the intention is they can be reopened, it would be helpful for the URL above to mention this - I was greatly confused because I was using exactly the method in that URL (as you can see above), it just turns out I was _unintentionally_ reopening them. (Perhaps reopening a closed client should issue a warning of its own?)

### To reproduce

See above.

### Expected behavior

I expect this to either work properly and not print a warning, or to fail, as described above.

### Actual behavior

The code works but prints a warning because it works accidentally, as described above.

### Debugging material

See above.

### Environment

macOS, Python 3.7.3, httpx 0.15.4 from pip, trio 0.17.0.
- OS: macOS
- Python version: 3.7.3
- HTTPX version: 0.15.4
- Async environment: Both, as indicated above. trio 0.17.0
- HTTP proxy: no
- Custom certificates: no

### Additional context

none


Discussion:
Or possibly `_is_closed` should be a tristate: `NOT_YET_OPENED` → `OPEN` → `CLOSED`
> Or possibly `_is_closed` should be a tristate: `NOT_YET_OPENED` → `OPEN` → `CLOSED`

That's the impression I had.

See https://github.com/encode/httpx/issues/1317 for other discussion.

This pull request removes the opening on send behaviour: https://github.com/encode/httpx/pull/1320 
Thanks for reporting this, @geofft! Indeed an interesting one.

This is somewhat related to #1317. tl;dr: indeed, the current `._is_closed` logic as of the latest release is broken, and #1320 aims at fixing that.

Could you install HTTPX from the branch on #1320 and see what's the new behavior your observe there? There's the idea of a separate `_seen_io` flag in the async case, which somewhat relates to njs' "triflag" idea.
Also interesting that this replicates fairly well on a raw HTTPCore application:

```python
import httpcore
import trio


async def get(transport, url, headers):
    print(f"Request {url}: started")
    _, _, stream, _ = await transport.arequest(b"GET", url, headers)
    body = b"".join([chunk async for chunk in stream]).decode("utf-8")
    print(f"Request {url}: finished")


async def main():
    async with trio.open_nursery() as nursery:
        async with httpcore.AsyncConnectionPool() as transport:
            headers = [(b"host", b"httpbin.org")]
            url1 = (b"https", b"httpbin.org", None, b"/delay/1")
            url2 = (b"https", b"httpbin.org", None, b"/delay/2")
            nursery.start_soon(get, transport, url1, headers)
            nursery.start_soon(get, transport, url2, headers)
        print("Transport closed")


if __name__ == "__main__":
    trio.run(main)
```

```console
$ python example.py
Transport closed
Request (b'https', b'httpbin.org', None, b'/delay/2'): started
Request (b'https', b'httpbin.org', None, b'/delay/1'): started
Request (b'https', b'httpbin.org', None, b'/delay/1'): finished
Request (b'https', b'httpbin.org', None, b'/delay/2'): finished
```
That is, even though we have exited the transport context by when requests actually fire off, it doesn't prevent those requests from being made, and doesn't raise any error. This means the connection pool is flushed, _then_ we make requests, potentially leaving connections hanging around until the program terminates.

Don't think that's what we're expecting to happen. 🤔 (I _think_ I'd expect the `transport.request()` to raise some kind of error saying "nope, you're not inside the transport context, you're doing something wrong".)
Actually, my intuition above^ is in line with what @tomchristie wrote in https://github.com/encode/httpx/issues/1317#issuecomment-698198191:

> I'd suggest we probably want the next proper version bump to hard-enforce that, raising an error if you attempt to send a request on a closed client.

But the tweak there is that this _might_ also be something we'd want to enforce at the transport level.
In the meantime, the workaround is to make sure the client is properly scope _above_ any kind of nursery / task group, so:

```python
import httpx
import asyncio

async def get(client, url):
    r = await client.get(url)
    print(r.text)

async def main():
    async with httpx.AsyncClient() as client:
        t1 = asyncio.create_task(get(client, "https://httpbin.org/delay/1"))
        t2 = asyncio.create_task(get(client, "https://httpbin.org/delay/2"))

        await t1
        await t2

if __name__ == "__main__":
    asyncio.run(main())
```

And:

```python
import httpx
import trio

async def get(client ,url):
    x = await client.get(url)
    print(x.text)

async def main():
    async with httpx.AsyncClient() as client:
        async with trio.open_nursery() as nursery:
            nursery.start_soon(get, client, "https://httpbin.org/delay/1")
            nursery.start_soon(get, client, "https://httpbin.org/delay/2")

if __name__ == "__main__":
    trio.run(main)
```
> Could you install HTTPX from the branch on #1320 and see what's the new behavior your observe there? There's the idea of a separate `_seen_io` flag in the async case, which somewhat relates to njs' "triflag" idea.

```
$ python3 -m venv httpx1320
$ httpx1320/bin/pip install -U pip
$ httpx1320/bin/pip install trio git+https://github.com/encode/httpx@refs/pull/1320/merge
$ httpx1320/bin/python aioproblem.py 
```

prints both requests but _not_ the warning, with either the asyncio or trio variants.

Re the other comments, hard error seems reasonable to me, and yes, swapping the order of the client scope and the task scope fixes things.

Resolving patch:
diff --git a/httpx/_client.py b/httpx/_client.py
--- a/httpx/_client.py
+++ b/httpx/_client.py
@@ -1,4 +1,5 @@
 import datetime
+import enum
 import functools
 import typing
 import warnings
@@ -71,6 +72,12 @@
 )
 
 
+class ClientState(enum.Enum):
+    UNOPENED = 1
+    OPENED = 2
+    CLOSED = 3
+
+
 class BaseClient:
     def __init__(
         self,
@@ -101,14 +108,14 @@ def __init__(
         }
         self._trust_env = trust_env
         self._netrc = NetRCInfo()
-        self._is_closed = True
+        self._state = ClientState.UNOPENED
 
     @property
     def is_closed(self) -> bool:
         """
         Check if the client being closed
         """
-        return self._is_closed
+        return self._state == ClientState.CLOSED
 
     @property
     def trust_env(self) -> bool:
@@ -750,8 +757,10 @@ def send(
 
         [0]: /advanced/#request-instances
         """
-        self._is_closed = False
+        if self._state == ClientState.CLOSED:
+            raise RuntimeError("Cannot send a request, as the client has been closed.")
 
+        self._state = ClientState.OPENED
         timeout = self.timeout if isinstance(timeout, UnsetType) else Timeout(timeout)
 
         auth = self._build_request_auth(request, auth)
@@ -1104,8 +1113,8 @@ def close(self) -> None:
         """
         Close transport and proxies.
         """
-        if not self.is_closed:
-            self._is_closed = True
+        if self._state != ClientState.CLOSED:
+            self._state = ClientState.CLOSED
 
             self._transport.close()
             for proxy in self._proxies.values():
@@ -1113,11 +1122,12 @@ def close(self) -> None:
                     proxy.close()
 
     def __enter__(self: T) -> T:
+        self._state = ClientState.OPENED
+
         self._transport.__enter__()
         for proxy in self._proxies.values():
             if proxy is not None:
                 proxy.__enter__()
-        self._is_closed = False
         return self
 
     def __exit__(
@@ -1126,13 +1136,12 @@ def __exit__(
         exc_value: BaseException = None,
         traceback: TracebackType = None,
     ) -> None:
-        if not self.is_closed:
-            self._is_closed = True
+        self._state = ClientState.CLOSED
 
-            self._transport.__exit__(exc_type, exc_value, traceback)
-            for proxy in self._proxies.values():
-                if proxy is not None:
-                    proxy.__exit__(exc_type, exc_value, traceback)
+        self._transport.__exit__(exc_type, exc_value, traceback)
+        for proxy in self._proxies.values():
+            if proxy is not None:
+                proxy.__exit__(exc_type, exc_value, traceback)
 
     def __del__(self) -> None:
         self.close()
@@ -1394,8 +1403,10 @@ async def send(
 
         [0]: /advanced/#request-instances
         """
-        self._is_closed = False
+        if self._state == ClientState.CLOSED:
+            raise RuntimeError("Cannot send a request, as the client has been closed.")
 
+        self._state = ClientState.OPENED
         timeout = self.timeout if isinstance(timeout, UnsetType) else Timeout(timeout)
 
         auth = self._build_request_auth(request, auth)
@@ -1750,8 +1761,8 @@ async def aclose(self) -> None:
         """
         Close transport and proxies.
         """
-        if not self.is_closed:
-            self._is_closed = True
+        if self._state != ClientState.CLOSED:
+            self._state = ClientState.CLOSED
 
             await self._transport.aclose()
             for proxy in self._proxies.values():
@@ -1759,11 +1770,12 @@ async def aclose(self) -> None:
                     await proxy.aclose()
 
     async def __aenter__(self: U) -> U:
+        self._state = ClientState.OPENED
+
         await self._transport.__aenter__()
         for proxy in self._proxies.values():
             if proxy is not None:
                 await proxy.__aenter__()
-        self._is_closed = False
         return self
 
     async def __aexit__(
@@ -1772,15 +1784,15 @@ async def __aexit__(
         exc_value: BaseException = None,
         traceback: TracebackType = None,
     ) -> None:
-        if not self.is_closed:
-            self._is_closed = True
-            await self._transport.__aexit__(exc_type, exc_value, traceback)
-            for proxy in self._proxies.values():
-                if proxy is not None:
-                    await proxy.__aexit__(exc_type, exc_value, traceback)
+        self._state = ClientState.CLOSED
+
+        await self._transport.__aexit__(exc_type, exc_value, traceback)
+        for proxy in self._proxies.values():
+            if proxy is not None:
+                await proxy.__aexit__(exc_type, exc_value, traceback)
 
     def __del__(self) -> None:
-        if not self.is_closed:
+        if self._state == ClientState.OPENED:
             warnings.warn(
                 f"Unclosed {self!r}. "
                 "See https://www.python-httpx.org/async/#opening-and-closing-clients "


Test patch:
diff --git a/tests/client/test_async_client.py b/tests/client/test_async_client.py
--- a/tests/client/test_async_client.py
+++ b/tests/client/test_async_client.py
@@ -4,6 +4,7 @@
 import pytest
 
 import httpx
+from tests.utils import MockTransport
 
 
 @pytest.mark.usefixtures("async_environment")
@@ -208,43 +209,39 @@ async def __aexit__(self, *args):
     ]
 
 
-@pytest.mark.usefixtures("async_environment")
-async def test_that_async_client_is_closed_by_default():
-    client = httpx.AsyncClient()
-
-    assert client.is_closed
+def hello_world(request):
+    return httpx.Response(200, text="Hello, world!")
 
 
 @pytest.mark.usefixtures("async_environment")
-async def test_that_send_cause_async_client_to_be_not_closed():
-    client = httpx.AsyncClient()
+async def test_client_closed_state_using_implicit_open():
+    client = httpx.AsyncClient(transport=MockTransport(hello_world))
 
+    assert not client.is_closed
     await client.get("http://example.com")
 
     assert not client.is_closed
-
     await client.aclose()
 
-
-@pytest.mark.usefixtures("async_environment")
-async def test_that_async_client_is_not_closed_in_with_block():
-    async with httpx.AsyncClient() as client:
-        assert not client.is_closed
+    assert client.is_closed
+    with pytest.raises(RuntimeError):
+        await client.get("http://example.com")
 
 
 @pytest.mark.usefixtures("async_environment")
-async def test_that_async_client_is_closed_after_with_block():
-    async with httpx.AsyncClient() as client:
-        pass
+async def test_client_closed_state_using_with_block():
+    async with httpx.AsyncClient(transport=MockTransport(hello_world)) as client:
+        assert not client.is_closed
+        await client.get("http://example.com")
 
     assert client.is_closed
+    with pytest.raises(RuntimeError):
+        await client.get("http://example.com")
 
 
 @pytest.mark.usefixtures("async_environment")
-async def test_that_async_client_caused_warning_when_being_deleted():
-    async_client = httpx.AsyncClient()
-
-    await async_client.get("http://example.com")
-
+async def test_deleting_unclosed_async_client_causes_warning():
+    client = httpx.AsyncClient(transport=MockTransport(hello_world))
+    await client.get("http://example.com")
     with pytest.warns(UserWarning):
-        del async_client
+        del client
diff --git a/tests/client/test_client.py b/tests/client/test_client.py
--- a/tests/client/test_client.py
+++ b/tests/client/test_client.py
@@ -4,6 +4,7 @@
 import pytest
 
 import httpx
+from tests.utils import MockTransport
 
 
 def test_get(server):
@@ -247,27 +248,29 @@ def __exit__(self, *args):
     ]
 
 
-def test_that_client_is_closed_by_default():
-    client = httpx.Client()
-
-    assert client.is_closed
+def hello_world(request):
+    return httpx.Response(200, text="Hello, world!")
 
 
-def test_that_send_cause_client_to_be_not_closed():
-    client = httpx.Client()
+def test_client_closed_state_using_implicit_open():
+    client = httpx.Client(transport=MockTransport(hello_world))
 
+    assert not client.is_closed
     client.get("http://example.com")
 
     assert not client.is_closed
+    client.close()
 
-
-def test_that_client_is_not_closed_in_with_block():
-    with httpx.Client() as client:
-        assert not client.is_closed
+    assert client.is_closed
+    with pytest.raises(RuntimeError):
+        client.get("http://example.com")
 
 
-def test_that_client_is_closed_after_with_block():
-    with httpx.Client() as client:
-        pass
+def test_client_closed_state_using_with_block():
+    with httpx.Client(transport=MockTransport(hello_world)) as client:
+        assert not client.is_closed
+        client.get("http://example.com")
 
     assert client.is_closed
+    with pytest.raises(RuntimeError):
+        client.get("http://example.com")