Repository: python-trio/trio

Issue #1614: Surviving fork()
Created: 2020-06-14T21:09:59Z
i ran into [the "no calling run inside run" error message] while testing with pytest-trio. i'm launching another process that is using trio using multiprocessing, but since i'm launching it from an async fixture, i get this error. i suppose the issue is that multiprocessing forks, so the context remains

Originally posted by @oakkitten in #608 (comment)
It would certainly be nicer if this worked! (Though note, there is a workaround for multiprocessing in the mean time: use set_start_method("spawn") or set_start_method("forkserver").)
So, what how can Trio survive if you call os.fork() inside a trio.run()? Which breaks down into two questions really: (1) how do we detect when a fork has happened? (2) once we've detected it, how do we clean things up in the nicest way possible?
Detection
The simplest approach is to save the value of os.getpid() when the runner is created, and then whenever we access the runner we check whether os.getpid() == the_saved_value. If our PID changes, then we're in a new process. This is what asyncio does.
Doing this everywhere that we access GLOBAL_RUN_STATE does add some overhead. (Contrary to what you may have heard, with recent glibc on Linux getpid() always does a full syscall, and syscalls have gotten more expensive in these days of meltdown/spectre/etc.)
The other option is to use os.register_at_fork. This totally removes the overhead. The downsides are that it's 3.7+ only. And, in theory, it's not quite as robust (it relies on the code that calls fork() also informing the Python interpreter that it has called fork()). But these limitations aren't too terrible. 3.6 is at the trailing edge of our support window, and we'll drop support within the next few years; in the mean time, telling folks who need fork() survival to upgrade to 3.7+ seems reasonable. And there are a lot of reasons why calling fork and then trying to continue running Python code will go badly if you don't tell the Python interpreter what you're doing; it's not just Trio that can break.
So my feeling is we should probably start with os.register_at_fork. And maybe consider adding a getpid() check at the top of trio.run as a just-in-case belt-and-suspenders kind of thing?
Cleaning up afterwards
So the obvious thing we need to do is to clear the child's GLOBAL_RUN_CONTEXT, so that Trio functions in the child won't try to mess with the parent's state.
There's also a question about what to do with the old run state. We could try to clean it up, by closing things. We could drop references to it, and let the GC try to clean it up. Or we could intentionally leak it.
Of these, I think leaking is really the only viable option... there are a few bits of state that we could plausibly clean up (e.g. closing the epoll/kqueue/IOCP handle), but the big thing is all the tasks and their stacks and any __del__ methods they might have. Trying to do explicit clean up on these will have unpredictable results, since it requires running arbitrary user code, that's definitely not expecting to be run inside a forked child. (I guess in theory we could reclaim the memory while avoiding running any destructors, but this would still leak all the descriptors etc., and anyway Python doesn't give us any way to do that.)
This is also why if you use fork() in a process with multiple threads, all the other thread stacks are just leaked: there's nothing else you can reasonably do with them.
So I guess what we want to do is just stash the old run state into a global somewhere, so it's pinned in memory until the process exits?
I'm actually not 100% sure how to reliably convince Python to never garbage collect an object graph. During shutdown, Python does try to collect module globals, and I'm not finding any great docs on that...
Maybe spawn a daemon thread that holds the pinned references on its stack, and then have it sleep forever? Weird gross hack but might work...

Duplicate issue #2127: func never called after trio.to_thread.run_sync(func, obj) (only for trio >= 0.16)
Created: 2021-09-22T08:09:20Z
I maintain a Python package https://foss.heptapod.net/fluiddyn/fluidimage using Trio which is working correctly with trio 0.15. Unfortunately, I started to have a buggy behavior with trio>=0.16.
A function given to trio.to_thread.run_sync seems to be never called, so that the program never ends. Of course, this happens only in particular situations that I was not able to reproduce with very simple code.
The situations in which there is the problem seem to be related with running the code multiple times first in the main process and then in other processes created with multiprocessing. These situations are mostly encountered in testing, but I had to avoid trio > 0.15, which is now a bit annoying for example to support newer versions of Python. So now I need to really understand what is the problem.
The call of trio.to_thread.run_sync which never lead to a call of the synchronous function looks like this (https://foss.heptapod.net/fluiddyn/fluidimage/-/blob/branch/default/fluidimage/executors/exec_async.py#L238)
    async def async_run_work_io(self, work):
        """Is destined to be started with a "trio.start_soon".
        obj = ...
        ret = await trio.to_thread.run_sync(work.func_or_cls, obj)


async_run_work_io is called in def_async_func_work_io (https://foss.heptapod.net/fluiddyn/fluidimage/-/blob/branch/default/fluidimage/executors/exec_async.py#L162)


def_async_func_work_io is called in start_async_works (https://foss.heptapod.net/fluiddyn/fluidimage/-/blob/branch/default/fluidimage/executors/exec_async.py#L97), where the nursery is opened.


I looked at https://trio.readthedocs.io/en/stable/history.html#trio-0-16-0-2020-06-10 to try to understand the issue. Some changes related to trio.to_thread.run_sync are mentioned but it does not help me to fix the problem.
Maybe you could have some clues?

Closing comment (2021-09-24T09:10:57Z): Gotcha. So this is almost certainly a bad interaction between fork and threads, similar to #1614. In particular, I think you're hitting a lack of fork-safety inside trio's thread cache. To confirm, try adding this janky workaround at the beginning of your program and see if it fixes your problem?
import os, trio

def afterfork():
    trio._core._thread_cache.THREAD_CACHE._idle_workers.clear()

os.register_at_fork(after_in_child=afterfork)
Also, I'm surprised you're not running into the other issues mentioned in #1614 – in particular, if you use multiprocessing in fork mode from inside trio.run, and then try to call trio.run again, that normally gives an error. I guess you must be using multiple calls to trio.run?