Repository: python-trio/trio

Issue:
TrioInternalError in trio 0.17 with sqlalchemy_aio
Issue reported by @mozesa in https://github.com/RazerM/sqlalchemy_aio/issues/35

Requirements:
```
pip install sqlalchemy_aio trio==0.17
```
The error does not happen on Trio 0.16.

To reproduce (happens ~50% of the time on my computer), and either very rarely or never with the `trio.sleep(0.1)`:
```python
from pathlib import Path

import trio
from sqlalchemy import Column, Integer, MetaData, Table, create_engine
from sqlalchemy.schema import CreateTable

from sqlalchemy_aio import TRIO_STRATEGY

db_file = Path("test.db")


async def main():
    db_file.unlink(missing_ok=True)
    engine = create_engine(f"sqlite:///{db_file}", strategy=TRIO_STRATEGY)

    metadata = MetaData()
    users = Table("users", metadata, Column("id", Integer, primary_key=True))

    await engine.execute(CreateTable(users))
    # await trio.sleep(0.1)  # seems to work if this is uncommented 


if __name__ == "__main__":
    trio.run(main)
```

Traceback:
```
Traceback (most recent call last):
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_run.py", line 2181, in unrolled_run
    runner.task_exited(task, final_outcome)
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_run.py", line 1506, in task_exited
    outcome.unwrap()
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/outcome/_sync.py", line 111, in unwrap
    raise captured_error
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_run.py", line 1599, in init
    run_sync_soon_nursery.cancel_scope.cancel()
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_run.py", line 815, in __aexit__
    raise combined_error_from_nursery
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_entry_queue.py", line 57, in kill_everything
    raise exc
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_entry_queue.py", line 53, in run_cb
    sync_fn(*args)
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_threads.py", line 282, in callback
    trio.lowlevel.spawn_system_task(await_in_trio_thread_task, name=afn)
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_generated_run.py", line 146, in spawn_system_task
    return GLOBAL_RUN_CONTEXT.runner.spawn_system_task(async_fn, *args, name=name)
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_run.py", line 1564, in spawn_system_task
    return self.spawn_impl(
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_run.py", line 1424, in spawn_impl
    raise RuntimeError("Nursery is closed to new arrivals")
RuntimeError: Nursery is closed to new arrivals

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/frazer/code/Python/sqlalchemy_aio/issue35.py", line 23, in <module>
    trio.run(main)
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_run.py", line 1919, in run
    timeout = gen.send(next_send)
  File "/Users/frazer/.pyenv/versions/sqlalchemy_aio/lib/python3.8/site-packages/trio/_core/_run.py", line 2226, in unrolled_run
    raise TrioInternalError("internal error in Trio - please file a bug!") from exc
trio.TrioInternalError: internal error in Trio - please file a bug!
```

---

If sqlalchemy_aio is doing something wrong it'll be in [sqlalchemy_aio/trio.py](https://github.com/RazerM/sqlalchemy_aio/blob/a8f9c72fe03d50cc8155681dffa89852978086f7/sqlalchemy_aio/trio.py)


Discussion:
Found the first failing commit with git bisect: https://github.com/python-trio/trio/commit/2947de7eb248e2f33790fb00bc7465490b0e0922
Hmm. Are you calling `trio.from_thread.run` from inside an async generator? It looks like the error is that somehow `trio.from_thread.run` is being called *very* late in trio's shutdown, after the function you passed to `trio.run` has already exited.

We should at least be handling this better... `from_thread.run` never considered the possibility of `spawn_system_task` failing, but that became possible after the async gens shutdown rework.

CC @oremanj
> Are you calling `trio.from_thread.run` from inside an async generator?

No
There's a one-tick interval where the system nursery is closed but we're still taking run_sync_soon callbacks. sqlalchemy_aio appears to use a system task that could reasonably wind up tickling this. I'm working on a fix.

Resolving patch:
diff --git a/trio/_core/_generated_run.py b/trio/_core/_generated_run.py
--- a/trio/_core/_generated_run.py
+++ b/trio/_core/_generated_run.py
@@ -126,6 +126,15 @@ def spawn_system_task(async_fn, *args, name=None):
 
         * System tasks do not inherit context variables from their creator.
 
+        Towards the end of a call to :meth:`trio.run`, after the main
+        task and all system tasks have exited, the system nursery
+        becomes closed. At this point, new calls to
+        :func:`spawn_system_task` will raise ``RuntimeError("Nursery
+        is closed to new arrivals")`` instead of creating a system
+        task. It's possible to encounter this state either in
+        a ``finally`` block in an async generator, or in a callback
+        passed to :meth:`TrioToken.run_sync_soon` at the right moment.
+
         Args:
           async_fn: An async callable.
           args: Positional arguments for ``async_fn``. If you want to pass
diff --git a/trio/_core/_run.py b/trio/_core/_run.py
--- a/trio/_core/_run.py
+++ b/trio/_core/_run.py
@@ -1546,6 +1546,15 @@ def spawn_system_task(self, async_fn, *args, name=None):
 
         * System tasks do not inherit context variables from their creator.
 
+        Towards the end of a call to :meth:`trio.run`, after the main
+        task and all system tasks have exited, the system nursery
+        becomes closed. At this point, new calls to
+        :func:`spawn_system_task` will raise ``RuntimeError("Nursery
+        is closed to new arrivals")`` instead of creating a system
+        task. It's possible to encounter this state either in
+        a ``finally`` block in an async generator, or in a callback
+        passed to :meth:`TrioToken.run_sync_soon` at the right moment.
+
         Args:
           async_fn: An async callable.
           args: Positional arguments for ``async_fn``. If you want to pass
diff --git a/trio/_threads.py b/trio/_threads.py
--- a/trio/_threads.py
+++ b/trio/_threads.py
@@ -1,3 +1,5 @@
+# coding: utf-8
+
 import threading
 import queue as stdlib_queue
 from itertools import count
@@ -248,7 +250,8 @@ def from_thread_run(afn, *args, trio_token=None):
 
     Raises:
         RunFinishedError: if the corresponding call to :func:`trio.run` has
-            already completed.
+            already completed, or if the run has started its final cleanup phase
+            and can no longer spawn new system tasks.
         Cancelled: if the corresponding call to :func:`trio.run` completes
             while ``afn(*args)`` is running, then ``afn`` is likely to raise
             :exc:`trio.Cancelled`, and this will propagate out into
@@ -279,7 +282,12 @@ async def unprotected_afn():
         async def await_in_trio_thread_task():
             q.put_nowait(await outcome.acapture(unprotected_afn))
 
-        trio.lowlevel.spawn_system_task(await_in_trio_thread_task, name=afn)
+        try:
+            trio.lowlevel.spawn_system_task(await_in_trio_thread_task, name=afn)
+        except RuntimeError:  # system nursery is closed
+            q.put_nowait(
+                outcome.Error(trio.RunFinishedError("system nursery is closed"))
+            )
 
     return _run_fn_as_system_task(callback, afn, *args, trio_token=trio_token)
 


Test patch:
diff --git a/trio/tests/test_threads.py b/trio/tests/test_threads.py
--- a/trio/tests/test_threads.py
+++ b/trio/tests/test_threads.py
@@ -7,6 +7,7 @@
 from .. import _core
 from .. import Event, CapacityLimiter, sleep
 from ..testing import wait_all_tasks_blocked
+from .._core.tests.tutil import buggy_pypy_asyncgens
 from .._threads import (
     to_thread_run_sync,
     current_default_thread_limiter,
@@ -554,3 +555,24 @@ def not_called():  # pragma: no cover
     trio_token = _core.current_trio_token()
     with pytest.raises(RuntimeError):
         from_thread_run_sync(not_called, trio_token=trio_token)
+
+
+@pytest.mark.skipif(buggy_pypy_asyncgens, reason="pypy 7.2.0 is buggy")
+def test_from_thread_run_during_shutdown():
+    save = []
+    record = []
+
+    async def agen():
+        try:
+            yield
+        finally:
+            with pytest.raises(_core.RunFinishedError), _core.CancelScope(shield=True):
+                await to_thread_run_sync(from_thread_run, sleep, 0)
+            record.append("ok")
+
+    async def main():
+        save.append(agen())
+        await save[-1].asend(None)
+
+    _core.run(main)
+    assert record == ["ok"]