Repository: dask/distributed

Issue:
Can't start worker on MacOS
On current master `distributed-2.9.1+13.g7d2ed43c` (and `tornado==6.0.3`), I get the following error on MacOS but not in Docker:
```
# already running dask scheduler
(.venv) ➜  model git:(nyc) ✗ PYTHONPATH=. dask-worker 'localhost:8786' --nthreads 1 --memory-limit 6GB --local-directory /tmp/ --nprocs 1
distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:55985'
objc[39479]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called.
objc[39479]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
distributed.nanny - INFO - Worker process 39479 was killed by signal 6
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x118220490>>, <Task finished coro=<Nanny._on_exit() done, defined at /Users/brett/model/.venv/lib/python3.7/site-packages/distributed/nanny.py:396> exception=TypeError('addresses should be strings or tuples, got None')>)
Traceback (most recent call last):
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/nanny.py", line 399, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/core.py", line 757, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/core.py", line 556, in send_recv
    raise exc.with_traceback(tb)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/core.py", line 408, in handle_comm
    result = handler(comm, **msg)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/scheduler.py", line 2122, in remove_worker
    address = self.coerce_address(address)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/scheduler.py", line 4831, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:55985'
distributed.dask_worker - INFO - End worker
```
No issues in the latest release (2.9.1). Possibly related to #3356 but I didn't see any similar error messages so I figured I'd keep it separate for now.


Discussion:
I have a similar setup on MacOS but I can't reproduce.  Can you also post your dask version ? 
Brett, if you have the time, this sounds like a good opportunity to use git
bisect

On Wed, Jan 15, 2020 at 11:35 AM Benjamin Zaitlen <notifications@github.com>
wrote:

> I have a similar setup on MacOS but I can't reproduce. Can you also post
> your dask version ?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/3374?email_source=notifications&email_token=AACKZTFLB5OBLV6KV66SGNLQ55QRNA5CNFSM4KHIKTX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJBRCWY#issuecomment-574820699>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTE6N3JSW2YGUT2CDN3Q55QRNANCNFSM4KHIKTXQ>
> .
>

Hmm, I'm also not able to reproduce on a fresh virtualenv so perhaps it's related to some other installed packages. I'll try something like a `bisect` on my `pip freeze` and see what I find :)
@mrocklin I did a bisect and found the [offending commit](https://github.com/dask/distributed/commit/32cb96effe9287fdc42dd73be36989be13abe99d): `32cb96effe9287fdc42dd73be36989be13abe99d is the first bad commit`.

```
# crashes
(.venv)  ➜  distributed git:(32cb96ef) ✗ dask-worker localhost:8786 --nanny
distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:52119'
distributed.diskutils - INFO - Found stale lock file and directory '/Users/brett/Dropbox/Documents/distributed/worker-g6fkk6vp', purging
objc[77514]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called.
objc[77514]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
distributed.nanny - INFO - Worker process 77514 was killed by signal 6
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:52119'
distributed.dask_worker - INFO - End worker

# go back one commit, no more crash
(.venv)  ➜  distributed git:(32cb96ef) ✗ gco @~1
Previous HEAD position was 32cb96ef Rework version checking (#2627)
HEAD is now at f68119a4 Add websocket scheduler plugin (#3335)
(.venv)  ➜  distributed git:(f68119a4) ✗ dask-worker localhost:8786 --nanny
distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:52128'
distributed.diskutils - INFO - Found stale lock file and directory '/Users/brett/Dropbox/Documents/distributed/worker-1r16h2w2', purging
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:52133
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:52133
distributed.worker - INFO -          dashboard at:            127.0.0.1:52134
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          8
distributed.worker - INFO -                Memory:                   17.18 GB
distributed.worker - INFO -       Local Directory: /Users/brett/Dropbox/Documents/distributed/worker-by5wbx39
distributed.worker - INFO - -------------------------------------------------
```
Still having trouble nailing down exactly which other packages are causing a conflict (sort of hard to really "bisect" when there's so many interdependencies between things) but maybe something about that commit rings a bell to someone..?
Update: there seems to be a bad interaction w/ bokeh..? But I still can't recreate on a fresh environment + distributed + bokeh so there's more to it than that 🤔 
```
(dask_test)  ➜  distributed git:(master) ✗ pip uninstall -y bokeh
Uninstalling bokeh-1.4.0:
  Successfully uninstalled bokeh-1.4.0
(dask_test)  ➜  distributed git:(master) ✗ /Users/brett/venvs/dask_test/bin/dask-worker localhost:8786
distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:59081'
distributed.diskutils - INFO - Found stale lock file and directory '/Users/brett/Dropbox/Documents/distributed/worker-7og5yzoa', purging
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:59082
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:59082
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
...



(dask_test)  ➜  distributed git:(master) ✗ pip install bokeh
Successfully installed bokeh-1.4.0

(dask_test)  ➜  distributed git:(master) ✗ /Users/brett/venvs/dask_test/bin/dask-worker localhost:8786
distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:59603'
distributed.diskutils - INFO - Found stale lock file and directory '/Users/brett/Dropbox/Documents/distributed/worker-ysqtkwgv', purging
objc[82717]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called.
objc[82717]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
distributed.nanny - INFO - Worker process 82717 was killed by signal 6
```
Interesting.  @bnaul can I ask you to try out https://github.com/dask/distributed/pull/3394 ?
Wait, nm, scratch that.  I got my comments crossed.
Found the offending line: https://github.com/dask/distributed/commit/32cb96effe9287fdc42dd73be36989be13abe99d#diff-14c5e43ed851ac4e461cda22da5340a8L24. Adding `pandas` back to this list fixes the issue for me on master.

Maybe something to do with the order of imports? I couldn't reproduce by stepping through in a debugger so it could also just be a very sensitive race condition...?

Also forgot to mention above but `--no-nanny` also avoids the issue
Wow, that's a weird thing to hang on.  

Can I ask you to add a `time.sleep(0.5)` in the `get_version` function and see if that would also make the problem go away?  Pandas is notable in that it takes a little bit of time to import.
(also, thanks for searching and finding that there)
I've been sprinkling some sleeps and can't seem to fix it but given the error message I assume it has to do with `set_forkserver_preload(preload)` where `preload` comes from `{required,optional}_packages`
@mrocklin any suggestions for a possible workaround for this? @quasiben if you'd like to try to reproduce, the following steps work for me on 3.7.6:
```
venvs/dask/bin/python3 -m venv venvs/dask_test
source venvs/dask_test/bin/activate
pip install numpy==1.18.1 Cython==0.29.14
pip install distributed pandas bokeh lz4 dask_ml blosc  # these are all the packages involved in the lines I posted above
venvs/dask_test/bin/dask-worker localhost:8786 --nanny
```
Full resulting package list:
```
blosc==1.8.3
bokeh==1.4.0
Click==7.0
cloudpickle==1.2.2
Cython==0.29.14
dask==2.9.2
dask-glm==0.2.0
dask-ml==1.2.0
distributed==2.9.3
HeapDict==1.0.1
Jinja2==2.11.0
joblib==0.14.1
llvmlite==0.31.0
lz4==3.0.2
MarkupSafe==1.1.1
msgpack==0.6.2
multipledispatch==0.6.0
numba==0.47.0
numpy==1.18.1
packaging==20.1
pandas==0.25.3
Pillow==7.0.0
psutil==5.6.7
pyparsing==2.4.6
python-dateutil==2.8.1
pytz==2019.3
PyYAML==5.3
scikit-learn==0.22.1
scipy==1.4.1
six==1.14.0
sortedcontainers==2.1.0
tblib==1.6.0
toolz==0.10.0
tornado==6.0.3
zict==1.0.0
```
Thanks for the continuing to investigate on this @bnaul! I installed the set of packages listed (copied the full list into a `requirements.txt` file and `pip install`ed them) in a fresh conda environment with Python 3.7.6, but wasn't able to reproduce the error. Here's the output I get after running `dask-worker localhost:8786 --nanny`:

<details>
<summary>dask worker output:</summary>

```
distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:64126'
distributed.diskutils - INFO - Found stale lock file and directory '/Users/jbourbeau/Desktop/worker-gkoh4zuf', purging
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:64127
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:64127
distributed.worker - INFO -          dashboard at:            127.0.0.1:64128
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    8.59 GB
distributed.worker - INFO -       Local Directory: /Users/jbourbeau/Desktop/worker-w7vu2nso
distributed.worker - INFO - -------------------------------------------------
```

</details>

We had a report about test failures on MacOS over in https://github.com/dask/distributed/issues/3356. Just a shot in the dark, but does @quasiben's comment over there (https://github.com/dask/distributed/issues/3356#issuecomment-574846558 and the following comment) apply to your system?
I don't believe that's related based on the errors. @jrbourbeau any chance you're able to try w/ a `homebrew` Python install instead of `conda`? That seems like a pretty big difference and likely to explain the discrepancy.
Still getting a similar output when using Python 3.7.6 from homebrew:

```
distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:55503'
distributed.diskutils - INFO - Found stale lock file and directory '/Users/jbourbeau/Desktop/worker-w7vu2nso', purging
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:55504
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:55504
distributed.worker - INFO -          dashboard at:            127.0.0.1:55505
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    8.59 GB
distributed.worker - INFO -       Local Directory: /Users/jbourbeau/Desktop/worker-0fw5zl57
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
distributed.worker - INFO - Waiting to connect to:       tcp://localhost:8786
```
@jrbourbeau are you on Catalina by any chance? I'm extending the experiment to any teammates within earshot and it seemed like MacOS version might be the determining factor: all of us still on Mojave are crashing, everyone on Catalina is fine 🤔 
Ah, I see! Yes, I am on Catalina
I am now able to reproduce.  I am on Mojave, I used brew to build a venv and pip installed same as @jrbourbeau .  I am also getting a lovely pop up with the following details:

<details>
Termination Reason:    Namespace OBJC, Code 0x1

Application Specific Information:
objc[74025]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called.
crashed on child side of fork pre-exec

Thread 0 Crashed:
0   libsystem_kernel.dylib        	0x00007fff77fee016 __abort_with_payload + 10
1   libsystem_kernel.dylib        	0x00007fff77fe95db abort_with_payload_wrapper_internal + 82
2   libsystem_kernel.dylib        	0x00007fff77fe9589 abort_with_reason + 22
3   libobjc.A.dylib               	0x00007fff766cf8dd _objc_fatalv(unsigned long long, unsigned long long, char const*, __va_list_tag*) + 108
4   libobjc.A.dylib               	0x00007fff766cf78f _objc_fatal(char const*, ...) + 135
5   libobjc.A.dylib               	0x00007fff766d060f performForkChildInitialize(objc_class*, objc_class*) + 341
6   libobjc.A.dylib               	0x00007fff766d162f initializeAndMaybeRelock(objc_class*, objc_object*, mutex_tt<false>&, bool) + 187
7   libobjc.A.dylib               	0x00007fff766c0690 lookUpImpOrForward + 228
8   libobjc.A.dylib               	0x00007fff766c0114 _objc_msgSend_uncached + 68
9   libobjc.A.dylib               	0x00007fff766c369b +[NSObject new] + 86
10  com.apple.Foundation          	0x00007fff4e18f478 -[NSThread init] + 61
11  com.apple.Foundation          	0x00007fff4e18f3e2 ____NSThreads_block_invoke + 64
12  libdispatch.dylib             	0x00007fff77e4e63d _dispatch_client_callout + 8
13  libdispatch.dylib             	0x00007fff77e4fd4b _dispatch_once_callout + 20
14  com.apple.Foundation          	0x00007fff4e18f39d _NSThreadGet0 + 325
15  com.apple.Foundation          	0x00007fff4e18ec31 _NSInitializePlatform + 407
16  libobjc.A.dylib               	0x00007fff766c1d51 call_load_methods + 233
17  libobjc.A.dylib               	0x00007fff766bf405 load_images + 117
18  dyld                          	0x000000011bdcb46a dyld::notifySingle(dyld_image_states, ImageLoader const*, 
</details>

`--no-nanny` also resolves the issue.  Additionally, using `spawn` for the ` multiprocessing-method` in `~/.config/dask/distributed.yaml` also resolves the problem

```yaml
  worker:
    multiprocessing-method: spawn
```

Digging a little more into brew and multiprocessing, it appears that this has been an issue in the past:
https://bugs.python.org/issue33725
@quasiben, I think that is because Apple made it so that [Grand Central Dispatch]( https://developer.apple.com/documentation/dispatch ) will error when forking (as it is not fork safe). This comes up commonly in the PyData ecosystem when libraries (like NumPy and SciPy) link against macOS's native BLAS library, [Accelerate]( https://developer.apple.com/documentation/accelerate ), which uses Grand Central Dispatch for threading. Spawning instead of forking fixes this issue. Alternatively using a different BLAS implementation (like OpenBLAS) avoids linking to Grand Central Dispatch thus also fixing this issue.
So @bnaul have you tried using spawn?
Yep, `spawn` works for me; if this is a MacOS problem then I guess it makes sense to close? Or maybe at least document somehow...? Will leave it up to you
Agree documentation is probably valuable. I wouldn't assume most users would know about this problem. Is this something you would be interested in writing up? 😉
We're seeing this forking issue when using `distributed.utils_test.client` in our unit tests. Any ideas for a fix?
@medley56 have you tried the workout outlined in https://github.com/dask/distributed/issues/3374#issuecomment-579060541
@quasiben Sorry, I meant is there any chance of a fix in distributed? After doing a bit more research though, it seems like the problem is deeper in MacOS and something to do with objective C's process forking being unsafe.

The workaround in #3374 seems to work.
Not sure what you mean by a fix in Distributed. As you noted, the OS is making this choice. More details in [this comment]( https://github.com/dask/distributed/issues/3374#issuecomment-579127093 ). So there’s not too much Distributed can really do unfortunately.
> Not sure what you mean by a fix in Distributed. As you noted, the OS is making this choice. More details in [this comment](https://github.com/dask/distributed/issues/3374#issuecomment-579127093). So there’s not too much Distributed can really do unfortunately.

I basically agree with this, but I would also point out that before [this](https://github.com/dask/distributed/commit/32cb96effe9287fdc42dd73be36989be13abe99d#diff-14c5e43ed851ac4e461cda22da5340a8L24) change this issue didn't occur for whatever mysterious reason. spawn is working for me but since forkserver is still the default, maybe it's worth thinking about whether there's a workaround? Documentation isn't really sufficient IMO if `distributed.Client()` just crashes right out of the gate for most Mac users...
So one solution would be to route forkserver -> spawn on OSX.  That is here: https://github.com/dask/distributed/pull/3460

Another solution would be to use spawn everywhere.  I'm toying around with this now.
Setting things to spawn actually makes the `client = Client()` experience a bit faster, down from 2s to 1.2s on my machine.  However it seems to make the `multiprocessing.Pipe`s that we use to detect broken connections somewhat noisy.

```python-traceback
Traceback (most recent call last):
  File "/home/mrocklin/miniconda/lib/python3.7/multiprocessing/queues.py", line 242, in _feed
    send_bytes(obj)
  File "/home/mrocklin/miniconda/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/mrocklin/miniconda/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/home/mrocklin/miniconda/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
```

Resolving patch:
diff --git a/distributed/nanny.py b/distributed/nanny.py
--- a/distributed/nanny.py
+++ b/distributed/nanny.py
@@ -597,6 +597,7 @@ async def kill(self, timeout=2, executor_wait=True):
                 "executor_wait": executor_wait,
             }
         )
+        await asyncio.sleep(0)  # otherwise we get broken pipe errors
         self.child_stop_q.close()
 
         while process.is_alive() and loop.time() < deadline:


Test patch:
diff --git a/distributed/deploy/tests/test_ssh.py b/distributed/deploy/tests/test_ssh.py
--- a/distributed/deploy/tests/test_ssh.py
+++ b/distributed/deploy/tests/test_ssh.py
@@ -38,14 +38,14 @@ async def test_keywords():
             "memory_limit": "2 GiB",
             "death_timeout": "5s",
         },
-        scheduler_options={"idle_timeout": "5s", "port": 0},
+        scheduler_options={"idle_timeout": "10s", "port": 0},
     ) as cluster:
         async with Client(cluster, asynchronous=True) as client:
             assert (
                 await client.run_on_scheduler(
                     lambda dask_scheduler: dask_scheduler.idle_timeout
                 )
-            ) == 5
+            ) == 10
             d = client.scheduler_info()["workers"]
             assert all(v["nthreads"] == 2 for v in d.values())