Repository: dask/distributed

Issue:
Error: Permission denied: 'primary_db.sqlite'
I have workers dying with the following error:

```
distributed.nanny - WARNING - Discovered failed worker
tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x2aaab987bb70>, <tornado.concurrent.Future object at 0x2aaab9876e10>)
Traceback (most recent call last):
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/ioloop.py", line 604, in _run_callback
    ret = callback()
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/ioloop.py", line 619, in <lambda>
    self.add_future(ret, lambda f: f.result())
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/rigel/home/ra2697/distributed/distributed/nanny.py", line 278, in _watch
    self.cleanup()
  File "/rigel/home/ra2697/distributed/distributed/nanny.py", line 258, in cleanup
    shutil.rmtree(self.worker_dir)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 418, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 418, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 418, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 418, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 418, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 418, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 438, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'primary_db.sqlite'
```


Discussion:
I haven't seen this before.  It looks like 

1.  The worker died
2.  The nanny tried to clean it up
3.  While removing the directory it tried to remove a sqlite file for which it didn't have permission.  
4.  This caused the nanny itself to stop functioning

I don't know what the sqlite file is from.  I take it this isn't something that you know about or added @rabernat ?  Perhaps an IPython kernel thing?  Were you using `start_ipython_workers`?

I don't know how to resolve the underlying error.  We could make the nanny robust to errors here though.  This is done here: https://github.com/dask/distributed/pull/1068
There is nothing custom about my environment. I don't see how ipython would be involved. The workers were started with the following command

``` bash
dask-worker --memory-limit 100e9 --nthreads 4 --local-directory /local \
       	--scheduler-file $HOME/.dask_schedule_file.json --interface ib0
```
([full slurm worker jobscript](https://github.com/rabernat/dask_distributed_demo/blob/master/jobscript_workers.sh))

Is there a way I can trace the underlying error which caused the workers to die? This is all I see in the worker logs. 
> Is there a way I can trace the underlying error which caused the workers to die? This is all I see in the worker logs.

There is no mechanism that I know of other than what we currently do.  We tend to log everything we can that is convenient.  You could increase logging verbosity in the config file to `debug`, but this may emit much more than you'd like to sift through.
I was able to get a little more traceback. This time some errors were passed to the client. I got this in my notebook.

```
distributed.utils - ERROR - ('open_dataset-c5ec3a7be5b583663504fa0a4d6ee23fiter-a4dc94731a56a449299d04153206cffe', 'tcp://10.43.8.135:40993')
Traceback (most recent call last):
  File "/rigel/home/ra2697/distributed/distributed/utils.py", line 212, in f
    result[0] = yield make_coro()
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/rigel/home/ra2697/distributed/distributed/client.py", line 1544, in _get
    result = yield self._gather(packed)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/rigel/home/ra2697/distributed/distributed/client.py", line 989, in _gather
    traceback)
  File "/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/six.py", line 686, in reraise
    raise value
distributed.scheduler.KilledWorker: ('open_dataset-c5ec3a7be5b583663504fa0a4d6ee23fiter-a4dc94731a56a449299d04153206cffe', 'tcp://10.43.8.135:40993')
---------------------------------------------------------------------------
KilledWorker                              Traceback (most recent call last)
<ipython-input-19-dc76bcde86bf> in <module>()
----> 1 get_ipython().magic("time weekly_iternum = ds_weekly['iter'].load()")

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/IPython/core/interactiveshell.py in magic(self, arg_s)
   2156         magic_name, _, magic_arg_s = arg_s.partition(' ')
   2157         magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2158         return self.run_line_magic(magic_name, magic_arg_s)
   2159 
   2160     #-------------------------------------------------------------------------

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line)
   2077                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2078             with self.builtin_trap:
-> 2079                 result = fn(*args,**kwargs)
   2080             return result
   2081 

<decorator-gen-59> in time(self, line, cell, local_ns)

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns)
   1183         else:
   1184             st = clock2()
-> 1185             exec(code, glob, local_ns)
   1186             end = clock2()
   1187             out = None

<timed exec> in <module>()

/rigel/home/ra2697/xarray/xarray/core/dataarray.py in load(self)
    573         working with many file objects on disk.
    574         """
--> 575         ds = self._to_temp_dataset().load()
    576         new = self._from_temp_dataset(ds)
    577         self._variable = new._variable

/rigel/home/ra2697/xarray/xarray/core/dataset.py in load(self)
    460 
    461             # evaluate all the dask arrays simultaneously
--> 462             evaluated_data = da.compute(*lazy_data.values())
    463 
    464             for k, data in zip(lazy_data, evaluated_data):

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/dask/base.py in compute(*args, **kwargs)
    200     dsk = collections_to_dsk(variables, optimize_graph, **kwargs)
    201     keys = [var._keys() for var in variables]
--> 202     results = get(dsk, keys, **kwargs)
    203 
    204     results_iter = iter(results)

/rigel/home/ra2697/distributed/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, **kwargs)
   1581         return sync(self.loop, self._get, dsk, keys, restrictions=restrictions,
   1582                     loose_restrictions=loose_restrictions,
-> 1583                     resources=resources)
   1584 
   1585     def _optimize_insert_futures(self, dsk, keys):

/rigel/home/ra2697/distributed/distributed/utils.py in sync(loop, func, *args, **kwargs)
    221         e.wait(1000000)
    222     if error[0]:
--> 223         six.reraise(*error[0])
    224     else:
    225         return result[0]

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/six.py in reraise(tp, value, tb)
    684         if value.__traceback__ is not tb:
    685             raise value.with_traceback(tb)
--> 686         raise value
    687 
    688 else:

/rigel/home/ra2697/distributed/distributed/utils.py in f()
    210                 raise RuntimeError("sync() called from thread of running loop")
    211             yield gen.moment
--> 212             result[0] = yield make_coro()
    213         except Exception as exc:
    214             logger.exception(exc)

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py in run(self)
   1013 
   1014                     try:
-> 1015                         value = future.result()
   1016                     except Exception:
   1017                         self.had_exception = True

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/concurrent.py in result(self, timeout)
    235             return self._result
    236         if self._exc_info is not None:
--> 237             raise_exc_info(self._exc_info)
    238         self._check_done()
    239         return self._result

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/util.py in raise_exc_info(exc_info)

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py in run(self)
   1019 
   1020                     if exc_info is not None:
-> 1021                         yielded = self.gen.throw(*exc_info)
   1022                         exc_info = None
   1023                     else:

/rigel/home/ra2697/distributed/distributed/client.py in _get(self, dsk, keys, restrictions, loose_restrictions, resources, raise_on_error)
   1542         packed = pack_data(keys, futures)
   1543         try:
-> 1544             result = yield self._gather(packed)
   1545         except Exception as e:
   1546             if raise_on_error:

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py in run(self)
   1013 
   1014                     try:
-> 1015                         value = future.result()
   1016                     except Exception:
   1017                         self.had_exception = True

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/concurrent.py in result(self, timeout)
    235             return self._result
    236         if self._exc_info is not None:
--> 237             raise_exc_info(self._exc_info)
    238         self._check_done()
    239         return self._result

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/util.py in raise_exc_info(exc_info)

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py in run(self)
   1019 
   1020                     if exc_info is not None:
-> 1021                         yielded = self.gen.throw(*exc_info)
   1022                         exc_info = None
   1023                     else:

/rigel/home/ra2697/distributed/distributed/client.py in _gather(self, futures, errors)
    987                             six.reraise(type(exception),
    988                                         exception,
--> 989                                         traceback)
    990                     if errors == 'skip':
    991                         bad_keys.add(key)

/rigel/home/ra2697/miniconda/envs/dask_distributed/lib/python3.5/site-packages/six.py in reraise(tp, value, tb)
    684         if value.__traceback__ is not tb:
    685             raise value.with_traceback(tb)
--> 686         raise value
    687 
    688 else:

KilledWorker: ('open_dataset-c5ec3a7be5b583663504fa0a4d6ee23fiter-a4dc94731a56a449299d04153206cffe', 'tcp://10.43.8.135:40993')
```
`KilledWorker` errors arise because enough workers have died while holding that task that the scheduler now suspects the task of causing difficulties.  
So guess this goes back to the underlying Permission error? What I don't get about that is it seems like it's coming from the nanny, not the worker itself. I really have no idea why the workers are dying.

Does anything that you do trigger the death?
Yes, they die when I try to compute stuff!

An abridged version of my workflow is the following:
```python
ds = xr.open_mfdataset(all_files, autoclose=False, decode_cf=False)
dsc = ds.chunk({'time': 1, 'k': 5, 'k_l': 5, 'k_u': 5, 'k_p1': 5})
ds_weekly = dsc.resample('7D', 'time')
# the following command causes some workers to die and raise the primary_db.sqlite error
weekly_iternum = ds_weekly['iter'].load()
```

Note that the amount of data being read is tiny! `iter` is a single interger value for each of the roughly ~2000 netcdf files in this dataset.

Does it fail when you use a subset of the files (maybe opening 2000 files is the issue?)

Does it fail if you do this on a single machine (maybe there is a software environment issue?)

Ideally we would be seeing nice errors on the workers when things fail.  Given that this isn't the case we may have to do some guess-and-checking.
This *does work* on a single multi-threaded worker.

It fails with two workers running on the same node with the same `Permission denied: 'primary_db.sqlite'` error (although this may not indicate the root cause of the worker dying).

edit: the total number of cores was the same in both cases. For the single worker, I specified 24 threads. For two workers, I gave each 12 threads.
Do you know what creates that file?  What permissions does it have?  As far as I know Dask never generates such a file.

Also, as of https://github.com/dask/distributed/pull/1068 that should just be a log message.  It shouldn't actually stop operation.
How about with a local cluster created with `Client()`
> Do you know what creates that file? What permissions does it have?

I have no idea what this file is either. As I said before, I have nothing weird or custom in my environment. I will try to figure it out though.
The permission error is coming from the nanny, when it tries to remove `worker_dir`:
https://github.com/dask/distributed/blob/master/distributed/nanny.py#L260
If you cna help me track down `worker_dir`, I can try to figure out what is going on.

Semi-good news! I finally got a more useful traceback from one of the workers.

```
HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 0:
  #000: H5T.c line 1716 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 0:
  #000: H5L.c line 1183 in H5Literate(): link iteration failed
    major: Symbol table
    minor: Iteration failed
  #001: H5Gint.c line 844 in H5G_iterate(): error iterating over links
    major: Symbol table
    minor: Iteration failed
  #002: H5Gobj.c line 693 in H5G__obj_iterate(): can't iterate over dense links
    major: Symbol table
    minor: Iteration failed
  #003: H5Gdense.c line 1073 in H5G__dense_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #004: H5Glink.c line 482 in H5G__link_iterate_table(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 0:
  #000: H5L.c line 1183 in H5Literate(): link iteration failed
    major: Symbol table
    minor: Iteration failed
  #001: H5Gint.c line 844 in H5G_iterate(): error iterating over links
    major: Symbol table
    minor: Iteration failed
  #002: H5Gobj.c line 693 in H5G__obj_iterate(): can't iterate over dense links
    major: Symbol table
    minor: Iteration failed
  #003: H5Gdense.c line 1073 in H5G__dense_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #004: H5Glink.c line 482 in H5G__link_iterate_table(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
distributed.protocol.pickle - INFO - Failed to deserialize b'\x80\x04\x95\xe9\x02\x00\x00\x00\x00\x00\x00\x8c\x14xarray.core.indexing\x94\x8c\x10CopyOnWriteArray\x94\x93\x94)\x81\x94}\x94(\x8c\x05array\x94h\x00\x8c\x12LazilyIndexedArray\x94\x93\x94)\x81\x94}\x94(\x8c\x03key\x94\x8c\x08builtins\x94\x8c\x05slice\x94\x93\x94NNN\x87\x94R\x94\x85\x94h\x05h\x07)\x81\x94}\x94(h\nh\rNNN\x87\x94R\x94\x85\x94h\x05\x8c\x18xarray.backends.netCDF4_\x94\x8c\x13NetCDF4ArrayWrapper\x94\x93\x94)\x81\x94}\x94(\x8c\x05dtype\x94\x8c\x05numpy\x94\x8c\x05dtype\x94\x93\x94\x8c\x02i8\x94K\x00K\x01\x87\x94R\x94(K\x03\x8c\x01<\x94NNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK\x00t\x94b\x8c\tdatastore\x94h\x16\x8c\x10NetCDF4DataStore\x94\x93\x94)\x81\x94}\x94(\x8c\tis_remote\x94\x89\x8c\x07_isopen\x94\x88\x8c\x05_mode\x94\x8c\x01r\x94\x8c\n_autoclose\x94\x89\x8c\x07_opener\x94\x8c\tfunctools\x94\x8c\x07partial\x94\x93\x94h\x16\x8c\x13_open_netcdf4_group\x94\x93\x94\x85\x94R\x94(h3\x8cK/rigel/ocp/users/ra2697/LLC/llc_4320_agulhas/llc_4320_agulhas.0000209808.nc\x94\x85\x94}\x94(\x8c\x08diskless\x94\x89\x8c\x06format\x94\x8c\x07NETCDF4\x94\x8c\x05group\x94N\x8c\x07clobber\x94\x88\x8c\x04mode\x94h,\x8c\x07persist\x94\x89uNt\x94b\x8c\x06writer\x94\x8c\x16xarray.backends.common\x94\x8c\x0bArrayWriter\x94\x93\x94)\x81\x94}\x94(\x8c\x07targets\x94]\x94\x8c\x07sources\x94]\x94ubh:h;\x8c\t_filename\x94h6ub\x8c\x05shape\x94K\x01\x85\x94\x8c\rvariable_name\x94\x8c\x04iter\x94ububub\x8c\x07_copied\x94\x89ub.'
Traceback (most recent call last):
  File "/rigel/home/ra2697/distributed/distributed/protocol/pickle.py", line 59, in loads
    return pickle.loads(x)
  File "/rigel/home/ra2697/xarray/xarray/backends/common.py", line 265, in __setstate__
    self.ds = self._opener(mode=self._mode)
  File "/rigel/home/ra2697/xarray/xarray/backends/netCDF4_.py", line 185, in _open_netcdf4_group
    ds = nc4.Dataset(filename, mode=mode, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 1848, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:13992)
OSError: NetCDF: HDF error
```

So it looks like the core problem has to do with netCDF and its underlying HDF library. This makes sense, since these libraries are the root of all evil in my life.
> If you can help me track down worker_dir, I can try to figure out what is going on.

```python
In [4]: client.run(lambda dask_worker: dask_worker.worker_dir, nanny=True)
Out[4]: 
{'tcp://127.0.0.1:32949': '/tmp/nanny-muje83fy',
 'tcp://127.0.0.1:39610': '/tmp/nanny-rcf30o58',
 'tcp://127.0.0.1:39732': '/tmp/nanny-w1a_1nft',
 'tcp://127.0.0.1:45843': '/tmp/nanny-6v8c408j'}
```
Right now I am just using a single worker (because I'm simultaneously actually trying to do some science 😉 )
Your command gives: `{'tcp://10.43.8.132:47460': '/local'}`

This is what I specified in my worker startup script
```bash
dask-worker --memory-limit 100e9 --nthreads 24 --local-directory /local \
       	--scheduler-file $HOME/.dask_schedule_file.json --interface ib0
```

In the case where this was failing, there would have been one worker with 4 threads per node. The directory `/local` is a local SSD scratch storage disk.

Could the problem simply be that dask is trying to remove the whole `/local` path? I could try specifying a sub-directory like `/local/dask-worker`. Does the path have to exist, or will dask create it if necessary?

Yes, that could be the problem.  Perhaps we should err if the directory exists, or perhaps we should create a subdirectory.  What behavior would you expect?
Here is what the docs from dask-worker say:
```
--local-directory TEXT        Directory to place worker files
```
My interpretation was that it expected something like `/tmp`. Now I understand that it should be something more specific, intended to be unique for each worker.

So yes, I think an error should be raised of the directory exists and the `dask-worker` documentation clarified.

I am going to do a test specifying `--local-directory /local/dask-worker-dir-$$`.
We could also create a new directory if the directory existed.  This would reduce the burden on people in your situation, who just want to specify `/local`
Either behavior is fine, as long as it is well documented.

From my perspective, it is easiest to just tell distributed where my local scratch storage is located, i.e. `/local`.
Ok, so I ran my test, and I can confirm that specifying `/local` was the problem. I gave it a sub-directory, and I no longer get the permission error. (Instead I get the _real_ error, related to netCDF...expect a new issue on that soon.)

Resolving patch:
diff --git a/distributed/cli/dask_worker.py b/distributed/cli/dask_worker.py
--- a/distributed/cli/dask_worker.py
+++ b/distributed/cli/dask_worker.py
@@ -79,8 +79,6 @@ def handle_signal(sig, frame):
               help="File to write the process PID")
 @click.option('--local-directory', default='', type=str,
               help="Directory to place worker files")
-@click.option('--temp-filename', default=None,
-              help="Internal use only")
 @click.option('--resources', type=str, default='',
               help='Resources for task constraints like "GPU=2 MEM=10e9"')
 @click.option('--scheduler-file', type=str, default='',
@@ -94,7 +92,7 @@ def handle_signal(sig, frame):
               help='Module that should be loaded by each worker process '
                    'like "foo.bar" or "/path/to/foo.py"')
 def main(scheduler, host, worker_port, http_port, nanny_port, nthreads, nprocs,
-         nanny, name, memory_limit, pid_file, temp_filename, reconnect,
+         nanny, name, memory_limit, pid_file, reconnect,
          resources, bokeh, bokeh_port, local_directory, scheduler_file,
          interface, death_timeout, preload, bokeh_prefix):
     if nanny:
@@ -189,18 +187,6 @@ def del_pid_file():
         if t is Nanny:
             global_nannies.append(n)
 
-    if temp_filename:
-        @gen.coroutine
-        def f():
-            while nannies[0].status != 'running':
-                yield gen.sleep(0.01)
-            import json
-            msg = {'port': nannies[0].port,
-                   'local_directory': nannies[0].local_dir}
-            with open(temp_filename, 'w') as f:
-                json.dump(msg, f)
-        loop.add_callback(f)
-
     @gen.coroutine
     def run():
         while all(n.status != 'closed' for n in nannies):
diff --git a/distributed/nanny.py b/distributed/nanny.py
--- a/distributed/nanny.py
+++ b/distributed/nanny.py
@@ -6,7 +6,6 @@
 from multiprocessing.queues import Empty
 import os
 import shutil
-import tempfile
 from time import sleep
 import weakref
 
@@ -46,17 +45,8 @@ def __init__(self, scheduler_ip, scheduler_port=None, worker_port=0,
         self.resources = resources
         self.death_timeout = death_timeout
         self.preload = preload
-        if not local_dir:
-            local_dir = tempfile.mkdtemp(prefix='nanny-')
-            self._should_cleanup_local_dir = True
-
-            @atexit.register
-            def _cleanup_local_dir():
-                if os.path.exists(local_dir):
-                    shutil.rmtree(local_dir)
-        else:
-            self._should_cleanup_local_dir = False
         self.local_dir = local_dir
+
         self.worker_dir = ''
         self.status = None
         self.process = None
@@ -264,8 +254,6 @@ def cleanup(self):
                         self.process.terminate()
 
     def __del__(self):
-        if self._should_cleanup_local_dir and os.path.exists(self.local_dir):
-            shutil.rmtree(self.local_dir)
         self.cleanup()
 
     @gen.coroutine
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -76,15 +76,18 @@ def __init__(self, scheduler_ip, scheduler_port=None, ncores=None,
             scheduler_addr = coerce_to_address((scheduler_ip, scheduler_port))
         self._port = 0
         self.ncores = ncores or _ncores
-        self.local_dir = local_dir or tempfile.mkdtemp(prefix='worker-')
         self.total_resources = resources or {}
         self.available_resources = (resources or {}).copy()
         self.death_timeout = death_timeout
         self.preload = preload
         if silence_logs:
             logger.setLevel(silence_logs)
-        if not os.path.exists(self.local_dir):
-            os.mkdir(self.local_dir)
+
+        if local_dir:
+            local_dir = os.path.abspath(local_dir)
+            if not os.path.exists(local_dir):
+                os.mkdir(local_dir)
+        self.local_dir = tempfile.mkdtemp(prefix='worker-', dir=local_dir)
 
         if memory_limit == 'auto':
             memory_limit = int(TOTAL_MEMORY * 0.6 * min(1, self.ncores / _ncores))


Test patch:
diff --git a/distributed/cli/tests/test_dask_worker.py b/distributed/cli/tests/test_dask_worker.py
--- a/distributed/cli/tests/test_dask_worker.py
+++ b/distributed/cli/tests/test_dask_worker.py
@@ -89,11 +89,13 @@ def test_local_directory(loop, nanny):
             with popen(['dask-worker', '127.0.0.1:8786', nanny,
                         '--no-bokeh', '--local-directory', fn]) as worker:
                 with Client('127.0.0.1:8786', loop=loop, timeout=10) as c:
+                    start = time()
                     while not c.scheduler_info()['workers']:
                         sleep(0.1)
+                        assert time() < start + 8
                     info = c.scheduler_info()
                     worker = list(info['workers'].values())[0]
-                    assert worker['local_directory'] == fn
+                    assert worker['local_directory'].startswith(fn)
 
 
 @pytest.mark.parametrize('nanny', ['--nanny', '--no-nanny'])
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -18,6 +18,7 @@
 from tornado.ioloop import TimeoutError
 
 import distributed
+from distributed import Nanny
 from distributed.core import rpc, connect
 from distributed.client import _wait
 from distributed.scheduler import Scheduler
@@ -722,3 +723,16 @@ def test_heartbeats(c, s, a, b):
     while not all(s.worker_info[w].get('memory-rss') for w in s.workers):
         yield gen.sleep(0.01)
         assert time() < start + 2
+
+
+@pytest.mark.parametrize('worker', [Worker, Nanny])
+def test_worker_dir(worker):
+    with tmpfile() as fn:
+        @gen_cluster(client=True, worker_kwargs={'local_dir': fn})
+        def test_worker_dir(c, s, a, b):
+            directories = [info['local_directory']
+                           for info in s.worker_info.values()]
+            assert all(d.startswith(fn) for d in directories)
+            assert len(set(directories)) == 2  # distinct
+
+        test_worker_dir()
diff --git a/distributed/utils_test.py b/distributed/utils_test.py
--- a/distributed/utils_test.py
+++ b/distributed/utils_test.py
@@ -24,7 +24,9 @@
 
 from .core import connect, rpc, CommClosedError
 from .metrics import time
+from .nanny import Nanny
 from .utils import ignoring, log_errors, sync, mp_context, get_ip, get_ipv6
+from .worker import Worker
 import pytest
 
 
@@ -479,8 +481,12 @@ def end_cluster(s, workers):
     def end_worker(w):
         with ignoring(TimeoutError, CommClosedError, EnvironmentError):
             yield w._close(report=False)
-        if w.local_dir and os.path.exists(w.local_dir):
-            shutil.rmtree(w.local_dir)
+        if isinstance(w, Nanny):
+            dir = w.worker_dir
+        else:
+            dir = w.local_dir
+        if dir and os.path.exists(dir):
+            shutil.rmtree(dir)
 
     yield [end_worker(w) for w in workers]
     yield s.close() # wait until scheduler stops completely