Repository: dask/dask

Issue:
Default `multiprocessing` context is broken and should never be used
The default `multiprocessing` context (on POSIX systems) is fork-without-exec. As a result, all process state is mirrored in the subprocesses, with often unexpected results.

1. Threads are not propagated. If the parent process started a thread and code relies on it working this can break the subprocess. This is probably fairly rare, though I do maintain a library where this is a problem.
2. More seriously, module-level globals are preserved. Common module-level state includes things like logging. Subprocesses inherit the parent process' logging, often with unexpected results. I have seen child processes trying to rotate the parent's log file, for example.
3. Even if dropping threads doesn't directly impact the subprocess, it can indirectly cause deadlocks. In particular, if it acquires a lock and a fork happens immediately after that, the lock will be acquired in the subprocess and *never released*.

The following example (on Python 3.6) demonstrates problems 2 and 3. Notice how merely using the `logging` module in a thread can break subsequent processes created with `multiprocessing`:

```python
import logging
import threading
import multiprocessing
import sys
import time

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.info("HELLO FROM PARENT")

# Emulate a thread that is holding the logging lock when fork() happens. In real
# code this would be a race condition, I'm just making it fail consistently for
# demonstration purposes.
def logging_thread():
    logging._acquireLock()
    for i in range(1, 10):
        print("THREAD COUNTDOWN TO RELEASE LOCK", i)
        time.sleep(1)
    logging._releaseLock()
    print("THREAD IS GOING AWAY")

t = threading.Thread(target=logging_thread)
t.start()


def child_process():
    logging.info("CHILD PROCESS 1")
    # Do something that acquires lock:
    logging.Handler().set_name("hello")
    logging.info("CHILD PROCESS 2")

p = multiprocessing.Process(target=child_process)
p.start()
```

Here is the output:

```console
$ python3.6 mp-fork.py 
INFO:root:HELLO FROM PARENT
THREAD COUNTDOWN TO RELEASE LOCK 1
INFO:root:CHILD PROCESS 1
THREAD COUNTDOWN TO RELEASE LOCK 2
THREAD COUNTDOWN TO RELEASE LOCK 3
THREAD COUNTDOWN TO RELEASE LOCK 4
THREAD COUNTDOWN TO RELEASE LOCK 5
THREAD COUNTDOWN TO RELEASE LOCK 6
THREAD COUNTDOWN TO RELEASE LOCK 7
THREAD COUNTDOWN TO RELEASE LOCK 8
THREAD COUNTDOWN TO RELEASE LOCK 9
THREAD IS GOING AWAY
```

"CHILD PROCESS 2" is never printed because the child process is deadlocked, and the process never exits.

---

I have seen processes mysteriously deadlock with default context, and start working just fine with `spawn` context. The deadlocks are impossible to debug, often, because they're highly complex race conditions.

Proposed solution: change default multiprocessing context to `spawn` on (Python 3 POSIX), which doesn't suffer from this problem. Tests can continue to run with default context for speed, given implementation of #3758. Python 2 will continue as is since it doesn't support the notion of contexts, Python 3 on Windows only supports `spawn` anyway.


Discussion:
Yeah bare forks are a nightmare. Improvements in this area are generally welcome. `spawn` works quite nicely. So it is reasonable to prefer that.

Another consideration would be to have proper fork+exec. IIUC `forkserver` does this (our current default). Also [Loky]( https://github.com/tomMoral/loky ) does this. Joblib version `0.12` uses Loky by default. There may be other relevant implementations out there.

Side note: Given the problems you pointed out, [this issue]( https://bugs.python.org/issue6721 ) may be interesting to you. There also appears to be an open PR ( https://github.com/python/cpython/pull/4071 ) to CPython to address the logging issues in particular.
Thanks for the pointers. Those would both improve the situation, but in the end it'll still be broken, it wouldn't fix locks in C libraries, or other module's global state.
How do you mean? Fork+exec overlays the existing process. There is nothing that sticks around.
Sorry, wasn't clear, I meant "the CPython issues/PRs specifically don't solve the underlying problem". fork+exec is the correct answer.
Ah ok. Yep, agreed.
It's worth noting that spawn is still probably better than fork+exec. [This benchmark from the Ruby community]( https://github.com/rtomayko/posix-spawn#benchmarks ) is pretty informative of the cost of fork+exec for large processes. Though it may differ for Python, would be surprised if it differed significantly. Especially given this problems with `fork` seem to show up in [C code on Linux]( https://bugzilla.redhat.com/show_bug.cgi?id=682922 ) and these [issues are discussed generally]( http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html ).

Resolving patch:
diff --git a/dask/multiprocessing.py b/dask/multiprocessing.py
--- a/dask/multiprocessing.py
+++ b/dask/multiprocessing.py
@@ -138,20 +138,16 @@ def pack_exception(e, dumps):
 
 def get_context():
     """ Return the current multiprocessing context."""
+    # fork context does fork()-without-exec(), which can lead to deadlocks,
+    # so default to "spawn".
+    context_name = config.get("multiprocessing.context", "spawn")
     if sys.platform == "win32":
-        # Just do the default, since we can't change it:
-        if config.get("multiprocessing.context", None) is not None:
+        if context_name != "spawn":
+            # Only spawn is supported on Win32, can't change it:
             warn(_CONTEXT_UNSUPPORTED, UserWarning)
         return multiprocessing
-    context_name = config.get("multiprocessing.context", None)
-    # The default context on OSX was switched from "fork" to "spawn" in
-    # Python 3.8. We keep "fork" as the default here for backwards
-    # compatibility and to avoid raising a RuntimeError when using the
-    # multiprocessing scheduler in a Python script that is not in a
-    # if __name__ == "__main__" block
-    if context_name is None and sys.platform == "darwin":
-        context_name = "fork"
-    return multiprocessing.get_context(context_name)
+    else:
+        return multiprocessing.get_context(context_name)
 
 
 def get(


Test patch:
diff --git a/dask/tests/test_multiprocessing.py b/dask/tests/test_multiprocessing.py
--- a/dask/tests/test_multiprocessing.py
+++ b/dask/tests/test_multiprocessing.py
@@ -194,15 +194,22 @@ def check_for_pytest():
 def test_custom_context_used_python3_posix():
     """ The 'multiprocessing.context' config is used to create the pool.
 
-    We assume default is 'fork', and therefore test for 'spawn'.  If default
-    context is changed this test will need to be modified to be different than
-    that.
+    We assume default is 'spawn', and therefore test for 'fork'.
     """
+    # We check for 'fork' by ensuring subprocess doesn't have modules only
+    # parent process should have:
+    def check_for_pytest():
+        import sys
+
+        return "FAKE_MODULE_FOR_TEST" in sys.modules
+
+    import sys
+
     sys.modules["FAKE_MODULE_FOR_TEST"] = 1
     try:
-        with dask.config.set({"multiprocessing.context": "spawn"}):
+        with dask.config.set({"multiprocessing.context": "fork"}):
             result = get({"x": (check_for_pytest,)}, "x")
-        assert not result
+        assert result
     finally:
         del sys.modules["FAKE_MODULE_FOR_TEST"]
 
@@ -215,19 +222,18 @@ def test_get_context_using_python3_posix():
 
     If default context is changed this test will need to change too.
     """
-    default_context = None if sys.platform != "darwin" else "fork"
-    assert get_context() is multiprocessing.get_context(default_context)
+    assert get_context() is multiprocessing.get_context("spawn")
     with dask.config.set({"multiprocessing.context": "forkserver"}):
         assert get_context() is multiprocessing.get_context("forkserver")
-    with dask.config.set({"multiprocessing.context": "spawn"}):
-        assert get_context() is multiprocessing.get_context("spawn")
+    with dask.config.set({"multiprocessing.context": "fork"}):
+        assert get_context() is multiprocessing.get_context("fork")
 
 
 @pytest.mark.skipif(sys.platform != "win32", reason="POSIX supports different contexts")
 def test_custom_context_ignored_elsewhere():
     """ On Windows, setting 'multiprocessing.context' doesn't explode.
 
-    Presumption is it's not used since unsupported, but mostly we care about
+    Presumption is it's not used since it's unsupported, but mostly we care about
     not breaking anything.
     """
     assert get({"x": (inc, 1)}, "x") == 2