Repository: dask/distributed

Issue:
Error closing a local cluster when client still running
**What happened**:

When a local cluster (using processes) shuts down, a ton of errors are now spewed about scheduling new futures after shutdown.

I can't replicate it in my distributed dev environment, but in a different environment (which is quite similar, also running dask & distributed from main—just py py3.9.1 instead of py3.9.5?) the process hangs and never terminates until a ctrl-C. In my distributed dev environment, the same errors are spewed, but it exits (with code 0 no the less).

`git bisect` implicates https://github.com/dask/distributed/pull/6031 @graingert.

**Minimal Complete Verifiable Example**:

```python
# repro.py
import distributed
from distributed.deploy.local import LocalCluster


if __name__ == "__main__":
    cluster = LocalCluster(n_workers=1, threads_per_worker=1, processes=True)
    client = distributed.Client(cluster)
```

```python-traceback
(dask-distributed) gabe dev/distributed ‹f4c52e9a› » python repro.py
2022-04-07 15:47:02,414 - distributed.utils - ERROR - cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 226, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1395, in _handle_report
    msgs = await self.scheduler_comm.comm.read()
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 242, in read
    convert_stream_closed_error(self, e)
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Client->Scheduler local=tcp://127.0.0.1:58800 remote=tcp://127.0.0.1:58791>: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/utils.py", line 693, in log_errors
    yield
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1225, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1254, in _ensure_connected
    comm = await connect(
  File "/Users/gabe/dev/distributed/distributed/comm/core.py", line 289, in connect
    comm = await asyncio.wait_for(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 439, in connect
    stream = await self.client.connect(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/site-packages/tornado/tcpclient.py", line 265, in connect
    addrinfo = await self.resolver.resolve(host, port, af)
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 424, in resolve
    for fam, _, _, _, address in await asyncio.get_running_loop().getaddrinfo(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/base_events.py", line 856, in getaddrinfo
    return await self.run_in_executor(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/base_events.py", line 814, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/concurrent/futures/thread.py", line 161, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2022-04-07 15:47:02,417 - distributed.utils - ERROR - cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 226, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1395, in _handle_report
    msgs = await self.scheduler_comm.comm.read()
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 242, in read
    convert_stream_closed_error(self, e)
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Client->Scheduler local=tcp://127.0.0.1:58800 remote=tcp://127.0.0.1:58791>: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/utils.py", line 693, in log_errors
    yield
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1401, in _handle_report
    await self._reconnect()
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1225, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1254, in _ensure_connected
    comm = await connect(
  File "/Users/gabe/dev/distributed/distributed/comm/core.py", line 289, in connect
    comm = await asyncio.wait_for(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 439, in connect
    stream = await self.client.connect(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/site-packages/tornado/tcpclient.py", line 265, in connect
    addrinfo = await self.resolver.resolve(host, port, af)
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 424, in resolve
    for fam, _, _, _, address in await asyncio.get_running_loop().getaddrinfo(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/base_events.py", line 856, in getaddrinfo
    return await self.run_in_executor(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/base_events.py", line 814, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/concurrent/futures/thread.py", line 161, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2022-04-07 15:47:02,471 - distributed.utils - ERROR - cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 226, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1395, in _handle_report
    msgs = await self.scheduler_comm.comm.read()
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 242, in read
    convert_stream_closed_error(self, e)
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Client->Scheduler local=tcp://127.0.0.1:58800 remote=tcp://127.0.0.1:58791>: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gabe/dev/distributed/distributed/utils.py", line 693, in log_errors
    yield
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1521, in _close
    await asyncio.wait_for(asyncio.shield(handle_report_task), 0.1)
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1401, in _handle_report
    await self._reconnect()
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1225, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/Users/gabe/dev/distributed/distributed/client.py", line 1254, in _ensure_connected
    comm = await connect(
  File "/Users/gabe/dev/distributed/distributed/comm/core.py", line 289, in connect
    comm = await asyncio.wait_for(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 439, in connect
    stream = await self.client.connect(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/site-packages/tornado/tcpclient.py", line 265, in connect
    addrinfo = await self.resolver.resolve(host, port, af)
  File "/Users/gabe/dev/distributed/distributed/comm/tcp.py", line 424, in resolve
    for fam, _, _, _, address in await asyncio.get_running_loop().getaddrinfo(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/base_events.py", line 856, in getaddrinfo
    return await self.run_in_executor(
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/base_events.py", line 814, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/concurrent/futures/thread.py", line 161, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
```

**Environment**:

- Dask version: 6e307662be633695f57bce79458b77e9be2e5702
- Python version: 3.9.5
- Operating System: macOS
- Install method (conda, pip, source): source


Discussion:
at python interpreter finalization:

* first `threading._shutdown()` which calls `concurrent.futures.thread._python_exit` and permanently shuts down all ThreadPoolExecutors: https://github.com/python/cpython/blob/f4b328e2bbbcc1096a28e903f07868b425397767/Lib/concurrent/futures/thread.py#L23-L37 https://bugs.python.org/issue39812
* then the at exit handlers run -  first atexit calls `distributed.deploy.spec.close_clusters` which waits 10 seconds for the LocalCuster to close
* meanwhile the client `handle_report_task` attempts to reconnect to the LocalCluster and cannot call `.resolve()` anymore because the thread pool is shutdown - note on tornado even though the host is already an IP address `resolve()` is still called using an executor https://github.com/tornadoweb/tornado/issues/3113
* then `atexit.register(_close_global_client)` gets called which collects the RuntimeError from the handle_report_task

The reason this worked before is that our ExecutorResolver used our legacy copy of ThreadPoolExecutor which uses atexit instead of threading._register_atexit: https://github.com/dask/distributed/blob/6e307662be633695f57bce79458b77e9be2e5702/distributed/_concurrent_futures_thread.py#L40-L50

A fix for this would be to ensure that clients then workers then clusters get shutdown atexit


I'm running into all sorts of problems regarding future/task scheduling and cancellation and blocking threadpools lately. Here are a couple of references but I can't tell you if any are actually related

- https://github.com/dask/distributed/pull/6064
- https://github.com/dask/distributed/pull/5910 
- https://github.com/dask/distributed/pull/6005
- https://github.com/dask/distributed/pull/5883 (had to be reverted but I saw cluster shutdowns being blocked due to a blocked event loop)
We should probably block release on this, yes?  cc @jrbourbeau 

@graingert is this something that you have time to own?  (this seems like the sort of thing that you know best among the team)

Resolving patch:
diff --git a/distributed/__init__.py b/distributed/__init__.py
--- a/distributed/__init__.py
+++ b/distributed/__init__.py
@@ -1,6 +1,7 @@
 from distributed import config  # isort:skip; load distributed configuration first
 from distributed import widgets  # isort:skip; load distributed widgets second
 
+import atexit
 
 import dask
 from dask.config import config  # type: ignore
@@ -73,3 +74,24 @@ def __getattr__(name):
         return __git_revision__
 
     raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+_python_shutting_down = False
+
+
+@atexit.register
+def _():
+    """Set a global when Python shuts down.
+
+    Note
+    ----
+    This function must be registered with atexit *after* any class that invokes
+    ``dstributed.utils.is_python_shutting_down`` has been defined. This way it
+    will be called before the ``__del__`` method of those classes.
+
+    See Also
+    --------
+    distributed.utils.is_python_shutting_down
+    """
+    global _python_shutting_down
+    _python_shutting_down = True
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -91,6 +91,7 @@
     format_dashboard_link,
     has_keyword,
     import_term,
+    is_python_shutting_down,
     log_errors,
     no_default,
     sync,
@@ -1231,6 +1232,7 @@ async def _reconnect(self):
                 except ImportError:
                     await self._close()
                     break
+
             else:
                 logger.error(
                     "Failed to reconnect to scheduler after %.2f "
@@ -1394,6 +1396,8 @@ async def _handle_report(self):
                     try:
                         msgs = await self.scheduler_comm.comm.read()
                     except CommClosedError:
+                        if is_python_shutting_down():
+                            return
                         if self.status == "running":
                             logger.info("Client report stream closed to scheduler")
                             logger.info("Reconnecting...")
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -1608,3 +1608,16 @@ def recursive_to_dict(
         return repr(obj)
     finally:
         tok.var.reset(tok)
+
+
+def is_python_shutting_down() -> bool:
+    """Is the interpreter shutting down now?
+
+    This is a variant of ``sys.is_finalizing`` which can return True inside the ``__del__``
+    method of classes defined inside the distributed package.
+    """
+    # This import must remain local for the global variable to be
+    # properly evaluated
+    from distributed import _python_shutting_down
+
+    return _python_shutting_down


Test patch: