Repository: dask/distributed

Issue:
Better handling of invalid local_directory
When copying `jobqueue.yaml` from one system to another, I missed updating `local_directory` to point to a valid path. When I tried to run a job, I got this output:

```
distributed.core - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/core.py", line 407, in handle_comm
    result = handler(comm, **msg)
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/scheduler.py", line 2146, in remove_worker
    address = self.coerce_address(address)
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/scheduler.py", line 4855, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
distributed.core - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/core.py", line 407, in handle_comm
    result = handler(comm, **msg)
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/scheduler.py", line 2146, in remove_worker
    address = self.coerce_address(address)
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/scheduler.py", line 4855, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
```

and then a hang. For a comparison, when using `distributed.LocalCluster` with an invalid `local_directory`, I get an infinite stream of this message:

```
Process Dask Worker process (from Nanny):
Traceback (most recent call last):
  File "/data/conda/envs/test/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/data/conda/envs/test/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/process.py", line 191, in _run
    target(*args, **kwargs)
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/nanny.py", line 675, in _run
    worker = Worker(**worker_kwargs)
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/worker.py", line 503, in __init__
    self._workspace = WorkSpace(os.path.abspath(local_directory))
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/diskutils.py", line 118, in __init__
    self._init_workspace()
  File "/data/conda/envs/test/lib/python3.8/site-packages/distributed/diskutils.py", line 124, in _init_workspace
    os.mkdir(self.base_dir)
FileNotFoundError: [Errno 2] No such file or directory: '/u/test/scratch'
```
I don't know dask, distributed, and dask_jobqueue well enough to say for sure, but I wonder if there is a way for more information about errors to be propagated back to dask_jobqueue from the remote workers to make problems like this easier to figure out?


Discussion:
Hi,

There is certainly room for improvement in the handling of this error and others! I've not looked into the code, but we can try to use your example to see what goes wrong.

Did you have the correct message in the worker logs, so the logs from the job where the worker was started?
Where could I access the worker logs? Are they just printed to stdout/stderr of the worker and I need to catch them from the job queuing system?
> Are they just printed to stdout/stderr of the worker and I need to catch them from the job queuing system?

Exactly. There were some discussion to retrieve them from the client at one point IIRC, but I don't know if this was pursued.

You can look at https://jobqueue.dask.org/en/latest/debug.html for other advices.
Just tried to reproduce this one, I think this is a distributed problem rather than dask-jobqueue.

Opposite to @willsALMANJ, I'm able to reproduce it with LocalCluster. Just doing:
```python
from distributed import LocalCluster
cluster = LocalCluster(local_directory='/some/inexistent/path')
```

Trigger a lot of errors:
```
distributed.utils - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/utils.py", line 656, in log_errors
    yield
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/scheduler.py", line 2214, in remove_worker
    address = self.coerce_address(address)
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/scheduler.py", line 4955, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
distributed.core - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/core.py", line 513, in handle_comm
    result = await result
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/scheduler.py", line 2214, in remove_worker
    address = self.coerce_address(address)
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/scheduler.py", line 4955, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <zmq.eventloop.ioloop.ZMQIOLoop object at 0x2b139447a210>>, <Task finished coro=<Nanny._on_exit() done, defined at /softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/nanny.py:440> exception=TypeError('addresses should be strings or tuples, got None')>)
Traceback (most recent call last):
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/nanny.py", line 443, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/core.py", line 660, in send_recv
    raise exc.with_traceback(tb)
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/core.py", line 513, in handle_comm
    result = await result
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/scheduler.py", line 2214, in remove_worker
    address = self.coerce_address(address)
  File "/softs/rh7/conda-envs/pangeo_202008/lib/python3.7/site-packages/distributed/scheduler.py", line 4955, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
```

I'm on Dask and distributed 2.2.0.

cc @TomAugspurger @jrbourbeau you may want to move this one to distributed repo.
Thanks for the ping @guillaumeeb, transferring over to distributed

Resolving patch:
diff --git a/distributed/nanny.py b/distributed/nanny.py
--- a/distributed/nanny.py
+++ b/distributed/nanny.py
@@ -438,7 +438,7 @@ def _on_exit_sync(self, exitcode):
         self.loop.add_callback(self._on_exit, exitcode)
 
     async def _on_exit(self, exitcode):
-        if self.status not in (Status.closing, Status.closed):
+        if self.status not in (Status.init, Status.closing, Status.closed):
             try:
                 await self.scheduler.unregister(address=self.worker_address)
             except (EnvironmentError, CommClosedError):


Test patch:
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -1647,3 +1647,20 @@ def bad_heartbeat_worker(*args, **kwargs):
                 else:
                     assert w.status == Status.closed
     assert "Heartbeat to scheduler failed" in logger.getvalue()
+
+
+@pytest.mark.asyncio
+async def test_bad_local_directory(cleanup):
+    async with await Scheduler() as s:
+        try:
+            async with Worker(s.address, local_directory="/not/a/valid-directory"):
+                pass
+        except PermissionError:
+            pass
+        else:
+            if WINDOWS:
+                pass
+            else:
+                assert False
+
+        assert not any("error" in log for log in s.get_logs())