Repository: agronholm/anyio

Issue:
`run_process` with `fail_after` causes RuntimeError on timeout
### Things to check first

- [X] I have searched the existing issues and didn't find my bug already reported there

- [X] I have checked that my bug is still present in the latest release


### AnyIO version

4.2.0

### Python version

3.12.1

### What happened?

I am using `fail_after` to timeout if some task group is taking too long, usually due to being stuck in some subprocess call longer than expected. I was able to reproduce this in a small script (posted in the other section) which results in the following exception after hitting the timeout handler in this case.

```
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x1011d8360>
Traceback (most recent call last):
  File "/Users/jmayeres/.pyenv/versions/3.12.1/lib/python3.12/asyncio/base_subprocess.py", line 126, in __del__
    self.close()
  File "/Users/jmayeres/.pyenv/versions/3.12.1/lib/python3.12/asyncio/base_subprocess.py", line 104, in close
    proto.pipe.close()
  File "/Users/jmayeres/.pyenv/versions/3.12.1/lib/python3.12/asyncio/unix_events.py", line 568, in close
    self._close(None)
  File "/Users/jmayeres/.pyenv/versions/3.12.1/lib/python3.12/asyncio/unix_events.py", line 592, in _close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/Users/jmayeres/.pyenv/versions/3.12.1/lib/python3.12/asyncio/base_events.py", line 792, in call_soon
    self._check_closed()
  File "/Users/jmayeres/.pyenv/versions/3.12.1/lib/python3.12/asyncio/base_events.py", line 539, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
```

The expectation would be that no RuntimeError is raised. I noticed that I don't have this issue if I use `open_process` and handle the output via text streams, but if I use `run_process` like in this example I get errors when the timeout is raised. I believe there's also some other weird stuff happening with these exceptions but I wanted to keep the example simple (in our working code there's some nested task groups and it seems like we don't get the timeout exception handler reached but the exception gets absorbed somewhere).

### How can we reproduce the bug?

Invoking the following in python results in an exception after printing "Timed out!". Note that I'm running this on macOS 14.2.1

```python
from anyio import create_task_group, fail_after, run, run_process


async def subprocess_output_no_stream():
    result = await run_process("while true; do ps aux; sleep 1; done")
    stdout = result.stdout
    print(f"Subprocess done: got {len(stdout)} bytes")


async def main():
    try:
        with fail_after(1):
            async with create_task_group() as tg:
                print("Starting")
                tg.start_soon(subprocess_output_no_stream)
            print("Task group done")
    except TimeoutError:
        print("Timed out!")


if __name__ == "__main__":
    run(main)
```

Comments:

Patch:
diff --git a/docs/versionhistory.rst b/docs/versionhistory.rst
index 5d39e45..db3dcb6 100644
--- a/docs/versionhistory.rst
+++ b/docs/versionhistory.rst
@@ -10,6 +10,18 @@ This library adheres to `Semantic Versioning 2.0 <http://semver.org/>`_.
 - Fixed passing ``total_tokens`` to ``anyio.CapacityLimiter()`` as a keyword argument
   not working on the ``trio`` backend
   (`#515 <https://github.com/agronholm/anyio/issues/515>`_)
+- Fixed ``Process.aclose()`` not performing the minimum level of necessary cleanup when
+  cancelled. Previously:
+
+  - Cancellation of ``Process.aclose()`` could leak an orphan process
+  - Cancellation of ``run_process()`` could very briefly leak an orphan process.
+  - Cancellation of ``Process.aclose()`` or ``run_process()`` on Trio could leave
+    standard streams unclosed
+
+  (PR by Ganden Schaffner)
+- Fixed ``Process.stdin.aclose()``, ``Process.stdout.aclose()``, and
+  ``Process.stderr.aclose()`` not including a checkpoint on asyncio (PR by Ganden
+  Schaffner)
 
 **4.2.0**
 
diff --git a/src/anyio/_backends/_asyncio.py b/src/anyio/_backends/_asyncio.py
index e884f56..2699bf8 100644
--- a/src/anyio/_backends/_asyncio.py
+++ b/src/anyio/_backends/_asyncio.py
@@ -918,6 +918,7 @@ class StreamReaderWrapper(abc.ByteReceiveStream):
 
     async def aclose(self) -> None:
         self._stream.feed_eof()
+        await AsyncIOBackend.checkpoint()
 
 
 @dataclass(eq=False)
@@ -930,6 +931,7 @@ class StreamWriterWrapper(abc.ByteSendStream):
 
     async def aclose(self) -> None:
         self._stream.close()
+        await AsyncIOBackend.checkpoint()
 
 
 @dataclass(eq=False)
@@ -940,14 +942,22 @@ class Process(abc.Process):
     _stderr: StreamReaderWrapper | None
 
     async def aclose(self) -> None:
-        if self._stdin:
-            await self._stdin.aclose()
-        if self._stdout:
-            await self._stdout.aclose()
-        if self._stderr:
-            await self._stderr.aclose()
-
-        await self.wait()
+        with CancelScope(shield=True):
+            if self._stdin:
+                await self._stdin.aclose()
+            if self._stdout:
+                await self._stdout.aclose()
+            if self._stderr:
+                await self._stderr.aclose()
+
+        try:
+            await self.wait()
+        except BaseException:
+            self.kill()
+            with CancelScope(shield=True):
+                await self.wait()
+
+            raise
 
     async def wait(self) -> int:
         return await self._process.wait()
diff --git a/src/anyio/_backends/_trio.py b/src/anyio/_backends/_trio.py
index 13b960f..1a47192 100644
--- a/src/anyio/_backends/_trio.py
+++ b/src/anyio/_backends/_trio.py
@@ -283,14 +283,21 @@ class Process(abc.Process):
     _stderr: abc.ByteReceiveStream | None
 
     async def aclose(self) -> None:
-        if self._stdin:
-            await self._stdin.aclose()
-        if self._stdout:
-            await self._stdout.aclose()
-        if self._stderr:
-            await self._stderr.aclose()
-
-        await self.wait()
+        with CancelScope(shield=True):
+            if self._stdin:
+                await self._stdin.aclose()
+            if self._stdout:
+                await self._stdout.aclose()
+            if self._stderr:
+                await self._stderr.aclose()
+
+        try:
+            await self.wait()
+        except BaseException:
+            self.kill()
+            with CancelScope(shield=True):
+                await self.wait()
+            raise
 
     async def wait(self) -> int:
         return await self._process.wait()
diff --git a/src/anyio/_core/_subprocesses.py b/src/anyio/_core/_subprocesses.py
index c161029..5d5d7b7 100644
--- a/src/anyio/_core/_subprocesses.py
+++ b/src/anyio/_core/_subprocesses.py
@@ -65,20 +65,18 @@ async def run_process(
         start_new_session=start_new_session,
     ) as process:
         stream_contents: list[bytes | None] = [None, None]
-        try:
-            async with create_task_group() as tg:
-                if process.stdout:
-                    tg.start_soon(drain_stream, process.stdout, 0)
-                if process.stderr:
-                    tg.start_soon(drain_stream, process.stderr, 1)
-                if process.stdin and input:
-                    await process.stdin.send(input)
-                    await process.stdin.aclose()
-
-                await process.wait()
-        except BaseException:
-            process.kill()
-            raise
+        async with create_task_group() as tg:
+            if process.stdout:
+                tg.start_soon(drain_stream, process.stdout, 0)
+
+            if process.stderr:
+                tg.start_soon(drain_stream, process.stderr, 1)
+
+            if process.stdin and input:
+                await process.stdin.send(input)
+                await process.stdin.aclose()
+
+            await process.wait()
 
     output, errors = stream_contents
     if check and process.returncode != 0:


Test patch:
diff --git a/tests/test_subprocesses.py b/tests/test_subprocesses.py
index 640384c..cc54fc5 100644
--- a/tests/test_subprocesses.py
+++ b/tests/test_subprocesses.py
@@ -8,8 +8,9 @@ from subprocess import CalledProcessError
 from textwrap import dedent
 
 import pytest
+from _pytest.fixtures import FixtureRequest
 
-from anyio import open_process, run_process
+from anyio import CancelScope, ClosedResourceError, open_process, run_process
 from anyio.streams.buffered import BufferedByteReceiveStream
 
 pytestmark = pytest.mark.anyio
@@ -176,3 +177,61 @@ async def test_run_process_inherit_stdout(capfd: pytest.CaptureFixture[str]) ->
     out, err = capfd.readouterr()
     assert out == "stdout-text" + os.linesep
     assert err == "stderr-text" + os.linesep
+
+
+async def test_process_aexit_cancellation_doesnt_orphan_process() -> None:
+    """
+    Regression test for #669.
+
+    Ensures that open_process.__aexit__() doesn't leave behind an orphan process when
+    cancelled.
+
+    """
+    with CancelScope() as scope:
+        async with await open_process(
+            [sys.executable, "-c", "import time; time.sleep(1)"]
+        ) as process:
+            scope.cancel()
+
+    assert process.returncode is not None
+    assert process.returncode != 0
+
+
+async def test_process_aexit_cancellation_closes_standard_streams(
+    request: FixtureRequest,
+    anyio_backend_name: str,
+) -> None:
+    """
+    Regression test for #669.
+
+    Ensures that open_process.__aexit__() closes standard streams when cancelled. Also
+    ensures that process.std{in.send,{out,err}.receive}() raise ClosedResourceError on a
+    closed stream.
+
+    """
+    if anyio_backend_name == "asyncio":
+        # Avoid pytest.xfail here due to https://github.com/pytest-dev/pytest/issues/9027
+        request.node.add_marker(
+            pytest.mark.xfail(reason="#671 needs to be resolved first")
+        )
+
+    with CancelScope() as scope:
+        async with await open_process(
+            [sys.executable, "-c", "import time; time.sleep(1)"]
+        ) as process:
+            scope.cancel()
+
+    assert process.stdin is not None
+
+    with pytest.raises(ClosedResourceError):
+        await process.stdin.send(b"foo")
+
+    assert process.stdout is not None
+
+    with pytest.raises(ClosedResourceError):
+        await process.stdout.receive(1)
+
+    assert process.stderr is not None
+
+    with pytest.raises(ClosedResourceError):
+        await process.stderr.receive(1)
diff --git a/tests/test_taskgroups.py b/tests/test_taskgroups.py
index f4c87b3..bc4a289 100644
--- a/tests/test_taskgroups.py
+++ b/tests/test_taskgroups.py
@@ -185,6 +185,9 @@ async def test_start_cancelled() -> None:
     assert not finished
 
 
+@pytest.mark.xfail(
+    sys.version_info < (3, 9), reason="Requires a way to detect cancellation source"
+)
 @pytest.mark.parametrize("anyio_backend", ["asyncio"])
 async def test_start_native_host_cancelled() -> None:
     started = finished = False
@@ -199,9 +202,6 @@ async def test_start_native_host_cancelled() -> None:
         async with create_task_group() as tg:
             await tg.start(taskfunc)
 
-    if sys.version_info < (3, 9):
-        pytest.xfail("Requires a way to detect cancellation source")
-
     task = asyncio.get_running_loop().create_task(start_another())
     await wait_all_tasks_blocked()
     task.cancel()