Repository: dask/distributed

Issue:
No locks available to Client
Hi again,

I'm having issues starting a client in the default local cluster mode:
```
(fdenv0) [stuckey@host ~]$ python
Python 3.6.2 |Anaconda custom (64-bit)| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask.distributed import Client
>>> client = Client()
Process ForkServerProcess-5:
Traceback (most recent call last):
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/process.py", line 175, in _run
    target(*args, **kwargs)
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/nanny.py", line 470, in _run
    worker = Worker(*worker_args, **worker_kwargs)
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/worker.py", line 1101, in __init__
    WorkerBase.__init__(self, *args, **kwargs)
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/worker.py", line 111, in __init__
    self._workdir = self._workspace.new_work_dir(prefix='worker-')
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/diskutils.py", line 219, in new_work_dir
    return WorkDir(self, **kwargs)
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/diskutils.py", line 47, in __init__
    with workspace._global_lock():
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/locket.py", line 194, in __enter__
    self.acquire()
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/locket.py", line 188, in acquire
    self._lock.acquire(self._timeout, self._retry_period)
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/locket.py", line 118, in acquire
    lock.acquire(timeout, retry_period)
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/locket.py", line 162, in acquire
    _lock_file_blocking(self._file)
  File "/opt/anaconda3/envs/fdenv0/lib/python3.6/site-packages/distributed/locket.py", line 52, in _lock_file_blocking
    fcntl.flock(file_.fileno(), fcntl.LOCK_EX)
OSError: [Errno 37] No locks available
Process ForkServerProcess-14:
...
```

My home directory, which holds the `dask-worker-space` sub-directory, is an NFSv3 network mount. However, the annoying thing is, it works fine on another presumably identical system.

Specifying the `local_dir` to a local directory works, so I will probably use that as a workaround:
```
>>> from dask.distributed import Client, LocalCluster
>>> worker_kwargs = {'local_dir': '/tmp'}
>>> cluster = LocalCluster(**worker_kwargs)
>>> client = Client(cluster)
>>> client
<Client: scheduler='tcp://127.0.0.1:41803' processes=28 cores=28>
```

Here are the relevant packages in my current environment:
```
cytoolz                   0.8.2                    py36_0    conda-forge
dask                      0.16.1                     py_0    conda-forge
dask-core                 0.16.1                     py_0    conda-forge
distributed               1.20.2                   py36_0    conda-forge
tornado                   4.5.3                    py36_0    conda-forge
```

Thanks for any help!


Discussion:
Interesting.  I think that @rabernat experienced a similar issue on a network file system.  It looks like we need to take a closer look at using file-based locks on network file systems.
Actually, googling the error message results in a possible solution that your NFS just may not be set up to provide file-based locks.  You might ask your system administrator.

Regardless, Dask should probably fail gracefully when it fails to acquire the lockfile.
Yep, we're looking into that now. Moving to NFSv4 may also solve the issue, though using a common `local_dir` as above is not a bad idea anyway.
By the way it looks like if you add the following line to your `~/.dask/config.yaml` file it will avoid file locking altogether

```yaml
use-file-locking: False
```

Resolving patch:
diff --git a/distributed/diskutils.py b/distributed/diskutils.py
--- a/distributed/diskutils.py
+++ b/distributed/diskutils.py
@@ -53,9 +53,18 @@ def __init__(self, workspace, name=None, prefix=None):
                 logger.debug("Locking %r...", self._lock_path)
                 # Avoid a race condition before locking the file
                 # by taking the global lock
-                with workspace._global_lock():
-                    self._lock_file = locket.lock_file(self._lock_path)
-                    self._lock_file.acquire()
+                try:
+                    with workspace._global_lock():
+                        self._lock_file = locket.lock_file(self._lock_path)
+                        self._lock_file.acquire()
+                except OSError as e:
+                    logger.exception("Could not acquire workspace lock on "
+                                     "path: %s ."
+                                     "Continuing without lock. "
+                                     "This may result in workspaces not being "
+                                     "cleaned up", self._lock_path,
+                                     exc_info=True)
+                    self._lock_file = None
             except Exception:
                 shutil.rmtree(self.dir_path, ignore_errors=True)
                 raise
@@ -89,7 +98,7 @@ def _finalize(cls, workspace, lock_path, lock_file, dir_path):
 class WorkSpace(object):
     """
     An on-disk workspace that tracks disposable work directories inside it.
-    If a process crash or another event left stale directories behind,
+    If a process crashes or another event left stale directories behind,
     this will be detected and the directories purged.
     """
 
@@ -219,5 +228,9 @@ def new_work_dir(self, **kwargs):
         name: str (optional)
             The subdirectory name for the workdir
         """
-        self._purge_leftovers()
+        try:
+            self._purge_leftovers()
+        except OSError:
+            logger.error("Failed to clean up lingering worker directories "
+                         "in path: %s ", exc_info=True)
         return WorkDir(self, **kwargs)


Test patch: