Repository: dask/distributed

Issue:
Python 3.9 compatibility
Opening this issue to track Python 3.9 compatibility issues.


Discussion:
Currently `distributed` fails to start worker processes on Python 3.9 with a local cluster (at least on osx). I see 

```python
In [3]: client = Client()
Task exception was never retrieved
future: <Task finished name='Task-396' coro=<_wrap_awaitable() done, defined at /opt/miniconda3/envs/prefect39/lib/python3.9/asyncio/tasks.py:678> exception=ImportError("cannot import name 'Popen' from partially initialized module 'multiprocessing.popen_spawn_posix' (most likely due to a circular import) (/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/popen_spawn_posix.py)")>
Traceback (most recent call last):
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/asyncio/tasks.py", line 685, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/core.py", line 305, in _
    await self.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 295, in start
    response = await self.instantiate()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 378, in instantiate
    result = await self.process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 575, in start
    await self.process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/context.py", line 283, in _Popen
    from .popen_spawn_posix import Popen
ImportError: cannot import name 'Popen' from partially initialized module 'multiprocessing.popen_spawn_posix' (most likely due to a circular import) (/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/popen_spawn_posix.py)
```
I get the same error in the python3.9 docker image (Debain 10 - Linux version 4.19.76) so I presume this is not limited to OSX
cc @itamarst this seems like something that might be up your alley
Always impressed at your nerd sniping skills :grin: 
I think that my main value to society today is indexing people by technical interest
Also, thank you for taking a look, I appreciate it

(I'm interpretting your :grin: as enthusiasm)
A workaround:

```python
>>> import multiprocessing.popen_spawn_posix
>>> from distributed import Client
>>> Client()
<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>
```

No idea what's going on yet, but I suspect it's a bug in Python.
Thank you for investigating this @itamarst 
Short term, would a solution be to add that import to the appropriate file within distributed?
I would guess so, yes (on POSIX platforms, anyway).
And a similar command line workaround:

`dask-worker --preload-nanny multiprocessing.popen_spawn_posix`

> A workaround:
> 
> ```python
> >>> import multiprocessing.popen_spawn_posix
> >>> from distributed import Client
> >>> Client()
> <Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>
> ```


Would you be interested in opening a PR Itamar?

Nice solution Lingfei Wang!

On Wed, Nov 4, 2020 at 4:30 PM Lingfei Wang <notifications@github.com>
wrote:

> And a similar command line workaround:
>
> dask-worker --preload-nanny multiprocessing.popen_spawn_posix
>
> A workaround:
>
> >>> import multiprocessing.popen_spawn_posix>>> from distributed import Client>>> Client()<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/4168#issuecomment-722049470>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTBDSD6NGWPZZ7G44SDSOHW2TANCNFSM4SRDXDTQ>
> .
>

hi all,

The workaround proposed doesn't seem to work on Windows / Python-3.9.1 / Distributed-2020.12.0 ... I notice it may only be ok for ["NOT WINDOWS" ](https://github.com/dask/distributed/pull/4234/files)

Any hope on this for non-windows, then Windows ? 
Any issue/fix identified/asked to cpython for next Python-3.9.2 I can follow ?

without the POSIXpatch
![image](https://user-images.githubusercontent.com/4312421/102718283-9cc58100-42e7-11eb-9978-1fb0623d103e.png)

with the POSIX patch (so not ok either)
![image](https://user-images.githubusercontent.com/4312421/102717793-ee204100-42e4-11eb-8109-07d5cd8c6fab.png)

hum, looking in bugs.cpython.org, maybe it's  this cpython issue  https://bugs.python.org/issue41567 , due to https://bugs.python.org/issue35943

if it's https://bugs.python.org/issue38884, Ansible defined a workaround: https://github.com/ansible/awx/pull/6093/files

For Windows workaround

```python
>>> from dask.distributed import Client
>>> import multiprocessing.popen_spawn_win32
>>> Client()
<Client: 'tcp://127.0.0.1:7182' processes=4 threads=8, memory=8.54 GB>

```
Thanks a lot @SunMaungOo , it works nicely.
> A workaround:
> 
> ```python
> >>> import multiprocessing.popen_spawn_posix
> >>> from distributed import Client
> >>> Client()
> <Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>
> ```
> 
> No idea what's going on yet, but I suspect it's a bug in Python.

Thanks @itamarst for the patch! Also solved my issue of `Client()` just getting stuck without any output on python 3.9.

Resolving patch:
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -64,6 +64,12 @@
 except ImportError:
     thread_state = threading.local()
 
+# For some reason this is required in python >= 3.9
+if WINDOWS:
+    import multiprocessing.popen_spawn_win32
+else:
+    import multiprocessing.popen_spawn_posix
+
 logger = _logger = logging.getLogger(__name__)
 
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -92,6 +92,7 @@
         "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
         "Topic :: Scientific/Engineering",
         "Topic :: System :: Distributed Computing",
     ],


Test patch:
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
--- a/.github/workflows/tests.yaml
+++ b/.github/workflows/tests.yaml
@@ -9,11 +9,12 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        # MacOS tests are currently afflicted by a very high level of flakiness. On
-        # Linux and Windows, run on merge to master and on all pull requests.
-        # On MacOS, run only on merge to master (see twin workflow tests-macos.yaml).
-        os: [ubuntu-latest, windows-latest]
-        python-version: ["3.6", "3.7", "3.8"]
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python-version: ["3.6", "3.7", "3.8", "3.9"]
+        exclude:
+          # Several failures; Python 3.6 to be phased out soon
+          - os: macos-latest
+            python-version: "3.6"
 
         # Uncomment to stress-test the test suite for random failures
         # This will take a LONG time and delay all PRs across the whole github.com/dask!
diff --git a/distributed/cli/tests/test_dask_worker.py b/distributed/cli/tests/test_dask_worker.py
--- a/distributed/cli/tests/test_dask_worker.py
+++ b/distributed/cli/tests/test_dask_worker.py
@@ -41,7 +41,7 @@ def test_nanny_worker_ports(loop):
                     if d["workers"]:
                         break
                     else:
-                        assert time() - start < 5
+                        assert time() - start < 60
                         sleep(0.1)
                 assert (
                     d["workers"]["tcp://127.0.0.1:9684"]["nanny"]
diff --git a/distributed/dashboard/tests/test_scheduler_bokeh.py b/distributed/dashboard/tests/test_scheduler_bokeh.py
--- a/distributed/dashboard/tests/test_scheduler_bokeh.py
+++ b/distributed/dashboard/tests/test_scheduler_bokeh.py
@@ -15,9 +15,10 @@
 import dask
 from dask.core import flatten
 from dask.utils import stringify
-from distributed.utils import format_dashboard_link
 from distributed.client import wait
+from distributed.compatibility import MACOS
 from distributed.metrics import time
+from distributed.utils import format_dashboard_link
 from distributed.utils_test import gen_cluster, inc, dec, slowinc, div, get_cert
 from distributed.dashboard.components.worker import Counters
 from distributed.dashboard.scheduler import applications
@@ -749,6 +750,7 @@ async def test_aggregate_action(c, s, a, b):
     assert ("compute") in mbk.action_source.data["names"]
 
 
+@pytest.mark.flaky(reruns=10, reruns_delay=5, condition=MACOS)
 @gen_cluster(client=True, scheduler_kwargs={"dashboard": True})
 async def test_compute_per_key(c, s, a, b):
     mbk = ComputePerKey(s)
diff --git a/distributed/deploy/tests/test_adaptive.py b/distributed/deploy/tests/test_adaptive.py
--- a/distributed/deploy/tests/test_adaptive.py
+++ b/distributed/deploy/tests/test_adaptive.py
@@ -302,7 +302,7 @@ async def test_adapt_down():
             start = time()
             while len(cluster.scheduler.workers) != 2:
                 await asyncio.sleep(0.1)
-                assert time() < start + 3
+                assert time() < start + 60
 
 
 @gen_test(timeout=30)
diff --git a/distributed/deploy/tests/test_spec_cluster.py b/distributed/deploy/tests/test_spec_cluster.py
--- a/distributed/deploy/tests/test_spec_cluster.py
+++ b/distributed/deploy/tests/test_spec_cluster.py
@@ -196,6 +196,7 @@ async def test_unexpected_closed_worker(cleanup):
             assert len(cluster.workers) == 2
 
 
+@pytest.mark.flaky(reruns=10, reruns_delay=5)
 @pytest.mark.slow
 @pytest.mark.asyncio
 async def test_restart(cleanup):
diff --git a/distributed/deploy/tests/test_ssh.py b/distributed/deploy/tests/test_ssh.py
--- a/distributed/deploy/tests/test_ssh.py
+++ b/distributed/deploy/tests/test_ssh.py
@@ -81,7 +81,7 @@ def test_defer_to_old(loop):
 
 
 @pytest.mark.avoid_ci
-def test_old_ssh_wih_local_dir(loop):
+def test_old_ssh_with_local_dir(loop):
     with pytest.warns(Warning):
         from distributed.deploy.old_ssh import SSHCluster as OldSSHCluster