Repository: dask/distributed

Issue:
Resource Warnings with LocalCluster 
Hi Everyone, 

After starting a local cluster on my machine as shown here:

```  cluster = LocalCluster()

    print(cluster)

    client = Client()

    print(client)

    cluster.close()
```

I got so many Resource Warnings as shown below. 

```/usr/local/lib/python3.6/dist-packages/distributed/diskutils.py:161:

 ResourceWarning:

unclosed file <_io.BufferedWriter name='/home/amal/PycharmProjects/dynamicparcels/dynpar/dask-worker-space/worker-4t43rjqe.dirlock'>

/usr/local/lib/python3.6/dist-packages/distributed/diskutils.py:161: ResourceWarning:

unclosed file <_io.BufferedWriter name='/home/amal/PycharmProjects/dynamicparcels/dynpar/dask-worker-space/worker-4fyxnn5d.dirlock'>

<Client: scheduler='tcp://127.0.0.1:43123' processes=8 cores=8>
distributed.scheduler - INFO - Clear task state
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:37009'
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:41811'
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:38581'
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:42381'
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:41167'
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:39899'
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:43027'
distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:34385'
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:39323
distributed.core - INFO - Removing comms to tcp://127.0.0.1:39323
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:46047
distributed.core - INFO - Removing comms to tcp://127.0.0.1:46047
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:45833
distributed.core - INFO - Removing comms to tcp://127.0.0.1:45833
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:36641
distributed.core - INFO - Removing comms to tcp://127.0.0.1:36641
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:42175
distributed.core - INFO - Removing comms to tcp://127.0.0.1:42175
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:33033
distributed.core - INFO - Removing comms to tcp://127.0.0.1:33033
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:46337
distributed.core - INFO - Removing comms to tcp://127.0.0.1:46337
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:37449
distributed.core - INFO - Removing comms to tcp://127.0.0.1:37449
distributed.scheduler - INFO - Lost all workers
distributed.scheduler - INFO - Scheduler closing...
distributed.scheduler - INFO - Scheduler closing all comms
/usr/lib/python3.6/asyncio/base_events.py:511: ResourceWarning:

unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
```

Do anyone have an idea how can I fix that please ?



Discussion:
cc @pitrou any thoughts on this?
Not really. The symptoms are rather vague...
Hi @bamal , 

have you tried to clean completely the folder? 

/home/amal/PycharmProjects/dynamicparcels/dynpar/dask-worker-space/

One possibility is that, since it looks like you're using pycharm, you still have running an underneath process or you stopped an unfinished process and one file is still locked. 
Are you using a private machine with admin privileges or a corporate one with less privileges?

I don't have a minimized reproducible case yet, but have also seen this warning occur repeatably when using instances of Client while running distributed 2021.02.0 under pytest 6.2, Python 3.8.8, Windows 7.

The warning is that an instance of asyncio.base_events.BaseEventLoop was not closed before being deleted by the garbage collector.  In the cases I've investigated the loops were created by calls beginning with Client.__init__, running through distributed.utils.LoopRunner.start, which creates a daemon thread that calls tornado's BaseAsyncIOLoop.start-->asyncio.get_event_loop->(distributed)AnyThreadLoopEventPolicy.get_event_loop->, which creates BaseEventLoop.  The loop then gets recorded in LoopRunner._all_loops.  It looks like it is meant to be closed by a call to LoopRunner.stop.

If anyone knows off-hand if the current behavior is intentional or not, that would be helpful to know before digging further into this.
@jonmoore the `LoopRunner` should be closed if the `Client` is closed, see https://github.com/dask/distributed/blob/67f57bc0c880101bc3336557c4dce43d4fa5ec43/distributed/client.py#L1489-L1490
I guess, for some reason, the client is not closed properly before the interpreter shuts down. We've been fixing a few stability related things over the past months and I would encourage you trying a more recent version to see if the problem persists.
@jonmoore, following up on @fjetter's comment, does upgrading to the latest `distributed` release help? 
See https://github.com/dask/distributed/pull/6122 and https://github.com/mwilliamson/locket.py/issues/14

Resolving patch:
diff --git a/distributed/diskutils.py b/distributed/diskutils.py
--- a/distributed/diskutils.py
+++ b/distributed/diskutils.py
@@ -9,9 +9,9 @@
 import weakref
 from typing import ClassVar
 
-import dask
+import locket
 
-from distributed import locket
+import dask
 
 logger = logging.getLogger(__name__)
 
diff --git a/distributed/locket.py b/distributed/locket.py
deleted file mode 100644
--- a/distributed/locket.py
+++ /dev/null
@@ -1,201 +0,0 @@
-# Vendored up-to-date copy of locket.py
-# Based on https://github.com/mwilliamson/locket.py/pull/8
-
-from __future__ import annotations
-
-import threading
-import weakref
-from time import sleep
-
-from distributed.metrics import time
-
-__all__ = ["lock_file"]
-
-
-try:
-    import fcntl
-except ImportError:
-    try:
-        import ctypes
-        import ctypes.wintypes
-        import msvcrt
-    except ImportError:
-        raise ImportError(
-            "Platform not supported (failed to import fcntl, ctypes, msvcrt)"
-        )
-    else:
-        _kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)  # type: ignore
-        _WinAPI_LockFile = _kernel32.LockFile
-        _WinAPI_LockFile.restype = ctypes.wintypes.BOOL
-        _WinAPI_LockFile.argtypes = [ctypes.wintypes.HANDLE] + [
-            ctypes.wintypes.DWORD  # type: ignore
-        ] * 4
-
-        _WinAPI_UnlockFile = _kernel32.UnlockFile
-        _WinAPI_UnlockFile.restype = ctypes.wintypes.BOOL
-        _WinAPI_UnlockFile.argtypes = [ctypes.wintypes.HANDLE] + [
-            ctypes.wintypes.DWORD  # type: ignore
-        ] * 4
-
-        _lock_file_blocking_available = False
-
-        def _lock_file_non_blocking(file_):
-            res = _WinAPI_LockFile(msvcrt.get_osfhandle(file_.fileno()), 0, 0, 1, 0)
-            if res:
-                return True
-            else:
-                err = ctypes.get_last_error()
-                # 33 = ERROR_LOCK_VIOLATION
-                if err != 33:
-                    raise ctypes.WinError(err)
-                return False
-
-        def _unlock_file(file_):
-            _WinAPI_UnlockFile(msvcrt.get_osfhandle(file_.fileno()), 0, 0, 1, 0)
-
-else:
-    _lock_file_blocking_available = True
-
-    def _lock_file_blocking(file_):
-        fcntl.flock(file_.fileno(), fcntl.LOCK_EX)
-
-    def _lock_file_non_blocking(file_):
-        try:
-            fcntl.flock(file_.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
-            return True
-        except (PermissionError, BlockingIOError):
-            return False
-
-    def _unlock_file(file_):
-        fcntl.flock(file_.fileno(), fcntl.LOCK_UN)
-
-
-_locks_lock = threading.Lock()
-_locks: weakref.WeakValueDictionary[str, _LockSet] = weakref.WeakValueDictionary()
-
-
-def lock_file(path, **kwargs):
-    with _locks_lock:
-        lock = _locks.get(path)
-        if lock is None:
-            lock = _create_lock_file(path)
-            _locks[path] = lock
-
-    return _Locker(lock, **kwargs)
-
-
-def _create_lock_file(path: str):
-    thread_lock = _ThreadLock(path)
-    file_lock = _LockFile(path)
-    return _LockSet([thread_lock, file_lock])
-
-
-class LockError(Exception):
-    pass
-
-
-def _acquire_non_blocking(acquire, timeout, retry_period, path):
-    if retry_period is None:
-        retry_period = 0.05
-
-    start_time = time()
-    while True:
-        success = acquire()
-        if success:
-            return
-        elif timeout is not None and time() - start_time > timeout:
-            raise LockError(f"Couldn't lock {path}")
-        else:
-            sleep(retry_period)
-
-
-class _LockSet:
-    def __init__(self, locks):
-        self._locks = locks
-
-    def acquire(self, timeout, retry_period):
-        acquired_locks = []
-        try:
-            for lock in self._locks:
-                lock.acquire(timeout, retry_period)
-                acquired_locks.append(lock)
-        except Exception:
-            for acquired_lock in reversed(acquired_locks):
-                # TODO: handle exceptions
-                acquired_lock.release()
-            raise
-
-    def release(self):
-        for lock in reversed(self._locks):
-            # TODO: Handle exceptions
-            lock.release()
-
-
-class _ThreadLock:
-    def __init__(self, path):
-        self._path = path
-        self._lock = threading.Lock()
-
-    def acquire(self, timeout=None, retry_period=None):
-        if timeout is None:
-            self._lock.acquire()
-        else:
-            _acquire_non_blocking(
-                acquire=lambda: self._lock.acquire(False),
-                timeout=timeout,
-                retry_period=retry_period,
-                path=self._path,
-            )
-
-    def release(self):
-        self._lock.release()
-
-
-class _LockFile:
-    def __init__(self, path):
-        self._path = path
-        self._file = None
-        self._thread_lock = threading.Lock()
-
-    def acquire(self, timeout=None, retry_period=None):
-        if self._file is None:
-            self._file = open(self._path, "wb")
-        if timeout is None and _lock_file_blocking_available:
-            _lock_file_blocking(self._file)
-        else:
-            _acquire_non_blocking(
-                acquire=lambda: _lock_file_non_blocking(self._file),
-                timeout=timeout,
-                retry_period=retry_period,
-                path=self._path,
-            )
-
-    def release(self):
-        _unlock_file(self._file)
-        self._file.close()
-        self._file = None
-
-
-class _Locker:
-    """
-    A lock wrapper to always apply the given *timeout* and *retry_period*
-    to acquire() calls.
-    """
-
-    def __init__(self, lock, timeout=None, retry_period=None):
-        self._lock = lock
-        self._timeout = timeout
-        self._retry_period = retry_period
-
-    def acquire(self):
-        self._lock.acquire(self._timeout, self._retry_period)
-
-    def release(self):
-        self._lock.release()
-
-    def __enter__(self):
-        self.acquire()
-        return self
-
-    def __exit__(self, exc_type, exc_value, traceback):
-        self.release()


Test patch:
diff --git a/distributed/tests/test_diskutils.py b/distributed/tests/test_diskutils.py
--- a/distributed/tests/test_diskutils.py
+++ b/distributed/tests/test_diskutils.py
@@ -173,7 +173,7 @@ def test_locking_disabled(tmpdir):
     base_dir = str(tmpdir)
 
     with dask.config.set({"distributed.worker.use-file-locking": False}):
-        with mock.patch("distributed.diskutils.locket.lock_file") as lock_file:
+        with mock.patch("locket.lock_file") as lock_file:
             assert_contents = functools.partial(assert_directory_contents, base_dir)
 
             ws = WorkSpace(base_dir)