Repository: dask/distributed

Issue:
ECSCluster raises error on exit for scheduler that has already stopped
**What happened**:

Using a Python REPL, if I create a Dask cluster using ECSCluster, wait for the scheduler to shut down from inactivity, then exit Python, an error is thrown by the exit handler because the scheduler is no longer reachable.

```python
>>> exit()
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7fb576979580>>, <Task finished name='Task-509' coro=<SpecCluster._correct_state_internal() done, defined at /usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py:325> exception=OSError('Timed out trying to connect to tcp://10.53.13.246:8786 after 10 s')>)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/distributed/comm/core.py", line 286, in connect
    comm = await asyncio.wait_for(
  File "/usr/local/lib/python3.8/asyncio/tasks.py", line 498, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/usr/local/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 406, in _close
    await self._correct_state()
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 333, in _correct_state_internal
    await self.scheduler_comm.retire_workers(workers=list(to_close))
  File "/usr/local/lib/python3.8/site-packages/distributed/core.py", line 788, in send_recv_from_rpc
    comm = await self.live_comm()
  File "/usr/local/lib/python3.8/site-packages/distributed/core.py", line 746, in live_comm
    comm = await connect(
  File "/usr/local/lib/python3.8/site-packages/distributed/comm/core.py", line 308, in connect
    raise IOError(
OSError: Timed out trying to connect to tcp://10.53.13.246:8786 after 10 s
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/distributed/comm/core.py", line 286, in connect
    comm = await asyncio.wait_for(
  File "/usr/local/lib/python3.8/asyncio/tasks.py", line 498, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 652, in close_clusters
    cluster.close(timeout=10)
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 110, in close
    return self.sync(self._close, callback_timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 189, in sync
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 340, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 324, in f
    result[0] = yield future
  File "/usr/local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/usr/local/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/usr/local/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 406, in _close
    await self._correct_state()
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 333, in _correct_state_internal
    await self.scheduler_comm.retire_workers(workers=list(to_close))
  File "/usr/local/lib/python3.8/site-packages/distributed/core.py", line 788, in send_recv_from_rpc
    comm = await self.live_comm()
  File "/usr/local/lib/python3.8/site-packages/distributed/core.py", line 746, in live_comm
    comm = await connect(
  File "/usr/local/lib/python3.8/site-packages/distributed/comm/core.py", line 308, in connect
    raise IOError(
OSError: Timed out trying to connect to tcp://10.53.13.246:8786 after 10 s
```

**What you expected to happen**:

The exit handler should ignore the already finished scheduler task, and continue with any other clean up as necessary.

**Minimal Complete Verifiable Example**:

```python
cluster = ECSCluster(…)

# wait for scheduler to exit from inactivity

exit()
```

**Anything else we need to know?**:

None.

**Environment**:

- Dask version: 2012.2.0
- Python version: 3.8.6
- Operating System: Debian (python:3.8 Docker image)
- Install method (conda, pip, source): pip


Discussion:
Thanks @kinghuang!

I'm going to move this over to distributed because I think this could affect all implementations of `SpecCluster`.

As part of the `SpecCluster` shutdown we attempt to gracefully stop workers via the Dask comm before we forcefully remove their resources. If the Dask comm is unavailable due to the scheduler exiting prematurely this exception is raised.

Resolving patch:
diff --git a/distributed/deploy/spec.py b/distributed/deploy/spec.py
--- a/distributed/deploy/spec.py
+++ b/distributed/deploy/spec.py
@@ -407,7 +407,7 @@ async def _close(self):
             for future in self._futures:
                 await future
             async with self._lock:
-                with suppress(CommClosedError):
+                with suppress(CommClosedError, OSError):
                     if self.scheduler_comm:
                         await self.scheduler_comm.close(close_workers=True)
                     else:


Test patch: