Repository: dask/distributed

Issue:
Client should call LoopRunner.stop()
**What happened**:
`distributed.Client` doesn't cleanup `LoopRunner`. Because of that `tornado.ioloop.IOLoop` and related threads remain running after closing the `Client`. The issue appears if run multiple clients with `asynchronous=False`. This happens because `Client` calls `LoopRunner.start()` but doesn't call `LoopRunner.stop()`.

**What you expected to happen**:
`distributed.Client` should stop `LoopRunner` to cleanup resources.

**Minimal Complete Verifiable Example**:

```python
import threading
import time
from distributed import LocalCluster, Client
from distributed.utils import LoopRunner


def main():
    threads_before = threading.enumerate()

    runner = LoopRunner(asynchronous=False)
    runner.start()

    cluster = LocalCluster(loop=runner.loop, asynchronous=False)
    client = Client(cluster, loop=runner.loop, asynchronous=False)

    # NOTE: uncomment to correctly cleanup resources.
    # client._should_close_loop = True

    client.close()
    cluster.close()

    runner.stop()

    # Wait until daemon threads finished.
    time.sleep(5)

    threads_after = threading.enumerate()

    if threads_before != threads_after:
        print('Before:', threads_before)
        print('After:', threads_after)
        raise AssertionError


if __name__ == '__main__':
    main()
```
If you set `client._should_close_loop = True` then it will call `LoopRunner.stop()`. Check https://github.com/dask/distributed/blob/master/distributed/client.py#L1435-L1436.

**Anything else we need to know?**:
I use old distributed/dask version, but the latest sources has the same code in this part. `LoopRunner` counts `start()` calls, so there must be the same number start/stop calls to correctly stop IO loop.

**Environment**:

- Dask version: 1.2.2
- Python version: 2.7.17 (Anaconda)
- Operating System: Linux Mint 18.3 Sylvia
- Install method (conda, pip, source): `conda/pip`



Discussion:
My guess is that Dask tries to only close down IOLoops that it created.  If you make an IOLoop externally and hand that to Dask then Dask won't try to close it because you might be using it for something else as well.

Also, just a small note, we've stopped supporting Python 2.7.
> then Dask won't try to close it ...

Yes, but it calls `LoopRunner.start()` which increments internal counter. This prevents parent `LoopRunner` from stopping `IOLoop`.

> ... because you might be using it for something else as well.

Well, `LoopRunner` handles this internally. My understanding is what calling start/stop is safe in this case.

* Bypass initialization and increment counter: https://github.com/dask/distributed/blob/master/distributed/utils.py#L397-L401
* Bypass deinitialization and decrement counter: https://github.com/dask/distributed/blob/master/distributed/utils.py#L461-L463

> Also, just a small note, we've stopped supporting Python 2.7.

Understand that, but it seems a version independent issue. Python 3 code is a bit different and just `client._should_close_loop = True` doesn't help.

Here `LocalCluster` calls  `LoopRunner.start()` in the `SpecCluster` constructor, but `LoopRunner.stop()` is called only in the `SpecCluster.__exit__()`, so if use `LocalCluster` directly, not as context manager, then again parent `LoopRunner` will not able to stop `IOLoop`.

* Start: https://github.com/dask/distributed/blob/master/distributed/deploy/spec.py#L274-L275
* Stop only in `__exit__()`: https://github.com/dask/distributed/blob/master/distributed/deploy/spec.py#L430, if call `close()` then `LoopRunner.stop()` is not called.

For py3  it cleanups correctly if run as:
```python
   ...
    with LocalCluster(loop=runner.loop, asynchronous=False) as cluster:
        client = Client(cluster, loop=runner.loop, asynchronous=False)
        client._should_close_loop = True
        client.close()
```

If I understand correctly when you call `LoopRunner.start()` then you must call `LoopRunner.stop()` otherwise `IOLoop` will remain running until shutdown. Currently Dask always calls `LoopRunner.start()` but `LoopRunner.stop()` is called only by some precondition. This may cause memory issues if you create `Cluster` or `Client` instances multiple times (number of running loops and related threads will grow).
Here is more advanced example demonstrating the issue, py3-based.

```python

import threading
import time
from distributed import LocalCluster, Client
from distributed.utils import LoopRunner


def print_list(title, lst):
    print('%s (num threads: %d):' % (title, len(lst)))
    for i, x in enumerate(lst):
        print('  %d: %s' % (i, x))


def run(cluster_runner, num_iter=3):
    for i in range(num_iter):
        print('Iteration %d/%d:' % (i + 1, num_iter))

        threads_before = threading.enumerate()

        loop_runner = LoopRunner(asynchronous=False)
        loop_runner.start()

        cluster_runner(loop_runner.loop)

        loop_runner.stop()

        # Allow IO loop process events and wait until daemon threads finished.
        time.sleep(1)

        threads_after = threading.enumerate()

        print_list('Threads Before', threads_before)
        print_list('Threads After', threads_after)


def direct(loop):
    cluster = LocalCluster(loop=loop, asynchronous=False)
    client = Client(cluster, loop=loop, asynchronous=False)
    # client._should_close_loop = True  # (un)comment
    time.sleep(1)  # Allow IO loop process events.
    client.close()
    cluster.close()


def context(loop):
    with LocalCluster(loop=loop, asynchronous=False) as cluster:
        client = Client(cluster, loop=loop, asynchronous=False)
        # client._should_close_loop = True  # (un)comment
        time.sleep(1)  # Allow IO loop process events.
        client.close()


def main():
    run(context)


if __name__ == '__main__':
    main()
```

Try `run(direct)` and `run(context)` with/without `client._should_close_loop = True`. You will see growing number of threads, except `run(context)` with `client._should_close_loop = True`.
(Sorry for the late reply)

Ok @skozlovf would you like to raise a PR with this change?
Ok, will create a PR for `Client` soon, but not sure about `LocalCluster` yet.

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -684,7 +684,6 @@ def __init__(
 
         self._connecting_to_scheduler = False
         self._asynchronous = asynchronous
-        self._should_close_loop = not loop
         self._loop_runner = LoopRunner(loop=loop, asynchronous=asynchronous)
         self.io_loop = self.loop = self._loop_runner.loop
 
@@ -1433,7 +1432,7 @@ async def _():
 
         assert self.status == "closed"
 
-        if self._should_close_loop and not shutting_down():
+        if not shutting_down():
             self._loop_runner.stop()
 
     async def _shutdown(self):
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -371,10 +371,8 @@ def __init__(self, loop=None, asynchronous=False):
                 # We're expecting the loop to run in another thread,
                 # avoid re-using this thread's assigned loop
                 self._loop = IOLoop()
-            self._should_close_loop = True
         else:
             self._loop = loop
-            self._should_close_loop = False
         self._asynchronous = asynchronous
         self._loop_thread = None
         self._started = False


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -5403,10 +5403,10 @@ async def test_client_name(s, a, b):
     await c.close()
 
 
-def test_client_doesnt_close_given_loop(loop, s, a, b):
-    with Client(s["address"], loop=loop) as c:
+def test_client_doesnt_close_given_loop(loop_in_thread, s, a, b):
+    with Client(s["address"], loop=loop_in_thread) as c:
         assert c.submit(inc, 1).result() == 2
-    with Client(s["address"], loop=loop) as c:
+    with Client(s["address"], loop=loop_in_thread) as c:
         assert c.submit(inc, 2).result() == 3
 
 
diff --git a/distributed/tests/test_client_loop.py b/distributed/tests/test_client_loop.py
new file mode 100644
--- /dev/null
+++ b/distributed/tests/test_client_loop.py
@@ -0,0 +1,34 @@
+import pytest
+from distributed import LocalCluster, Client
+from distributed.utils import LoopRunner
+
+
+# Test if Client stops LoopRunner on close.
+@pytest.mark.parametrize("with_own_loop", [True, False])
+def test_close_loop_sync(with_own_loop):
+    loop_runner = loop = None
+
+    # Setup simple cluster with one threaded worker.
+    # Complex setup is not required here since we test only IO loop teardown.
+    cluster_params = dict(n_workers=1, dashboard_address=None, processes=False)
+
+    loops_before = LoopRunner._all_loops.copy()
+
+    # Start own loop or use current thread's one.
+    if with_own_loop:
+        loop_runner = LoopRunner()
+        loop_runner.start()
+        loop = loop_runner.loop
+
+    with LocalCluster(loop=loop, **cluster_params) as cluster:
+        with Client(cluster, loop=loop) as client:
+            client.run(max, 1, 2)
+
+    # own loop must be explicitly stopped.
+    if with_own_loop:
+        loop_runner.stop()
+
+    # Internal loops registry must the same as before cluster running.
+    # This means loop runners in LocalCluster and Client correctly stopped.
+    # See LoopRunner._stop_unlocked().
+    assert loops_before == LoopRunner._all_loops