Repository: dask/distributed

Issue:
quiet cancelledError for scheduler shutdown
`gen.with_timeout` unfortunately permanently adds a callback to futures, and when the timeout is over, it doesn't remove that callback.

with python asyncio futures, if you get the result of them when they are cancelled they raise `CancelledErrors`.....  This isn't caught by `error_callback` in `with_timeout`, and so it is printed.

we cancel the futures a few lines down.

for reference: (the tornado callback)

```
    def error_callback(future):
        try:
            future.result()
        except Exception as e:
            if not isinstance(e, quiet_exceptions):
                app_log.error("Exception in Future %r after timeout",
                              future, exc_info=True)
```

possible fix for https://github.com/dask/distributed/issues/2273

Spurious CancelledError on LocalCluster.close()
The following repro produces a spurious log error from tornado/asyncio on closing the LocalCluster:

```
[00:36:18.413 ERROR  ] Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
  File "../lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback
    future.result()
  File "../lib/python3.5/asyncio/futures.py", line 286, in result
    raise CancelledError
concurrent.futures._base.CancelledError
```

Here is the repro:

```
from joblib import parallel_backend

import distributed.joblib
from distributed import Client, LocalCluster


def run_example():
    lc = LocalCluster(n_workers=1)
    client = Client(lc)
    with parallel_backend('dask.distributed', scheduler_host=client.scheduler.address):
        pass
    client.close()
    lc.close()


if __name__ == "__main__":
    run_example()
```

Couldn't figure out how to silence or what exactly is wrong here. Appreciate any guidance.

Distributed versions is 1.23.1 on python 3.5.6 and joblib 0.12.3

```
dask                      0.19.1                   py35_0  
dask-core                 0.19.1                   py35_0  
tornado                   5.1              py35h14c3975_0 
```



Discussion:

I'm getting a slightly different error locally (although I'm also on more recent versions):

```pytz
future: <Future finished exception=CommClosedError('in <closed TCP>: Stream is closed',)>
Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 177, in read
    n_frames = yield stream.read_bytes(8)
  File "/home/mrocklin/workspace/tornado/tornado/gen.py", line 1133, in run
    value = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mrocklin/workspace/tornado/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 198, in read
    convert_stream_closed_error(self, e)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
```

In this case we have some coroutine waiting on a comm.  That comm gets broken as part of the shutdown process in an unclean way.   The way to resolve this is to ensure that we close down all comms cleanly during shutdown.  If I were to try debugging this I would probably try to find out which comm was not being shut down well (possibly by recording the last message it received) and then tracking the lifetime of that comm to see if/when a close message gets sent to it.  

This is an example PR that fixes a similar problem: https://github.com/dask/distributed/pull/2237
I'm getting similar errors as OP, but I'm using distributed, independent workers. The main, i.e. the process that generates the submits to the workers, has the error

```
2018-10-30 10:32:36,659 : TME gen tornado.application.error_callback +974: ERROR    IO loop: :- : Exception in Future
Traceback (most recent call last):
  File "Lib\site-packages\tornado\gen.py", line 970, in error_callback
concurrent.futures._base.CancelledError
```

This is with python 3.6.7 anaconda
```
dask==0.19.4
dask-core==0.19.4
distributed==1.23.3
tornado==5.1.1
```

I see dask & dask-core 0.20.0 is released, so I will re-check with that version.
I have updated 
```
dask==0.20.0
dask-core==0.20.0
distributed==1.24.0
tornado==5.1.1
```

and still see the same issues.
I was also getting the same error message. I couldn't figure out how to stop the error message so I manually closed the client before exiting and caught the exception.

```python
from concurrent.futures._base import CancelledError

...

try:
    client.close()
except CancelledError:
    print('Dask distributed processing did not shut down cleanly.')
```

I am seeing this at exit when my script sets up a LocalCluster. Apparently the operations are performed.
Is there an guarantee that the workers have flushed all data when the cluster is torn down even the final close down sees this error?

@ericness solution works for me to remove this message
The workaround did not work for me to remove this message for a LocalCluster.

I spent some time looking at this one, here is a way of reproducing the problem using only `distributed`. The following uses distributed master (it seems that distributed 1.25.2 has the same behaviour but I did not try older versions).

```py
# test-distributed.py
import time

from distributed import Client, LocalCluster


def func(x):
    time.sleep(1)


if __name__ == '__main__':
    cluster = LocalCluster()
    client = Client(cluster)

    # I don't think you need to .submit + .gather to trigger the errors below 
    # but this was intended to leave some time for the workers to come online
    future = client.submit(func, 1)
    client.gather(future)
    # cluster.close is called first
    cluster.close()
    client.close()
```

I run the script with `python test-distributed.py` (in an interactive session, you only see these errors when you exit the session e.g. when you do Control-D in IPython).

I get two kind of errors:

```
Exception ignored in: <generator object add_client at 0x7f137884c938>
RuntimeError: generator ignored GeneratorExit
tornado.application - ERROR - Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
  File "/home/lesteve/miniconda3/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
concurrent.futures._base.CancelledError
```

```
tornado.application - ERROR - Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
  File "/home/lesteve/miniconda3/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
concurrent.futures._base.CancelledError
Future exception was never retrieved
future: <Future finished exception=CommClosedError('in <distributed.comm.tcp.TCPConnector object at 0x7f37a001a128>: ConnectionRefuse
Traceback (most recent call last):
  File "/home/lesteve/dev/distributed/distributed/comm/tcp.py", line 342, in connect
    **kwargs)
  File "/home/lesteve/miniconda3/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/lesteve/miniconda3/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/lesteve/miniconda3/lib/python3.6/site-packages/tornado/tcpclient.py", line 232, in connect
    af, addr, stream = yield connector.start(connect_timeout=timeout)
  File "/home/lesteve/miniconda3/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/lesteve/miniconda3/lib/python3.6/site-packages/tornado/tcpclient.py", line 112, in on_connect_done
    stream = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lesteve/miniconda3/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/lesteve/dev/distributed/distributed/comm/tcp.py", line 353, in connect
    convert_stream_closed_error(self, e)
  File "/home/lesteve/dev/distributed/distributed/comm/tcp.py", line 129, in convert_stream_closed_error
    raise CommClosedError("in %s: %s: %s" % (obj, exc.__class__.__name__, exc))
distributed.comm.core.CommClosedError: in <distributed.comm.tcp.TCPConnector object at 0x7f37a001a128>: ConnectionRefusedError: [Errn
```

Something I noticed is when `client.close` before `cluster.close`, I only get warnings (which are a bit less noisy and roughly 60% of the time).
 
```py
import time

from distributed import Client, LocalCluster


def func(x):
    time.sleep(1)


if __name__ == '__main__':
    cluster = LocalCluster()
    client = Client(cluster)

    future = client.submit(func, 1)
    result = client.gather(future)
    # client.close is called first
    client.close()
    cluster.close()
```

Here are the warnings: 
```
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:58848 remote=tcp://127.0.0.1:44667>
```

Final note for some context: I was not explicitly calling `client.close` or `cluster.close` in my original script and was getting the errors. My understanding is that since `.cluster` is an attribute of `client`, `cluster` will be deleted first at the end of the python session. I am guessing that behind the scenes that causes `cluster.close` to be called before `client.close` behind the scenes.
Adding the following:

```python
import atexit
atexit.register(client.close)
```

also makes it impossible to collect cluster (and therefore closing it) without having closed the client first.

The problem is that this prevents garbage collection of the client to happen before the end of the Python process, which is suboptimal.

I have tried to register a `weakref.finalize` on the cluster object to close the client first but the finalizer seems to be called after `cluster.__del__`  and therefore `cluster.close` which is too late.

Another option would be to maintain a `weakref.WeakSet` of clients that reference a given cluster on the cluster itself and make sure to close the client in `cluster.close`. However that would only work for subclasses of the `Cluster` class and not the `SLURMCluster` class that is a subclass of dask-jobqueue `ClusterManager` instead.
I seem to be experiencing a similar problem, except with a distributed cluster instead of local:
```python
import time

import dask.distributed

def work(x):
    time.sleep(1)
    return x

if __name__ == '__main__':
    client = dask.distributed.Client('myhost:8786')
    future = client.submit(work, 42)
    result = client.gather(future)
    print(f'Result: {result}')
```
Output:
```
Result: 42
tornado.application - ERROR - Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
  File "/Users/flynn/code/whisper/venv/lib/python3.6/site-packages/tornado/gen.py", line 970, in error_callback
    future.result()
concurrent.futures._base.CancelledError
```
Otherwise everything else works as intended. Explicitly closing the client or wrapping it in a try/except doesn't help (exception seems to happen somewhere in the background)

I'm on macOS 10.14.2 with Python 3.6.5 and Dask installed via pip:
```
dask==1.0.0
distributed==1.25.2
tornado==5.1.1
```
> Otherwise everything else works as intended.

Same experience: besides confusion, I did not experience any negative side-effects. I would completely understand if this is not deemed high priority, especially given that @ogrisel's investigation seems to indicate that this is a non-trivial fix.

Another work-around I had that may work for you is to set the `tornado.application` logger level to `FATAL`. Caveat: you may silence other useful errors if you do that.

Interestingly this error is not visible when using a Jupyter notebook so this may be why this was not noticed and fixed until now (it makes sense that the stdout/stderr after the python kernel closes does not reach the notebook UI).

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -1090,7 +1090,8 @@ def _close(self, fast=False):
             # This makes the shutdown slightly smoother and quieter
             with ignoring(AttributeError, gen.TimeoutError):
                 yield gen.with_timeout(timedelta(milliseconds=100),
-                                       self._handle_scheduler_coroutine)
+                                       self._handle_scheduler_coroutine,
+                                       quiet_exceptions=(CancelledError,))
 
             if self.scheduler_comm and self.scheduler_comm.comm and not self.scheduler_comm.comm.closed():
                 yield self.scheduler_comm.close()


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -4579,6 +4579,22 @@ def test_quiet_client_close(loop):
             ), line
 
 
+def test_quiet_client_close_when_cluster_is_closed_before_client(loop):
+    n_attempts = 5
+    # Trying a few times to reduce the flakiness of the test. Without the bug
+    # fix in #2477 and with 5 attempts, this test passes by chance in about 10%
+    # of the cases.
+    for _ in range(n_attempts):
+        with captured_logger(logging.getLogger('tornado.application')) as logger:
+            cluster = LocalCluster(loop=loop)
+            client = Client(cluster, loop=loop)
+            cluster.close()
+            client.close()
+
+        out = logger.getvalue()
+        assert 'CancelledError' not in out
+
+
 @gen_cluster()
 def test_close(s, a, b):
     c = yield Client(s.address, asynchronous=True)