Repository: dask/distributed

Issue:
SpecCluster doesn't clean up after failed scheduler start
As noted in multiple cluster managers based on `SpecCluster` in dask/dask-cloudprovider#262 if connectivity to the scheduler fails when creating the scheduler resource the object isn't cleaned up properly. This can happen when there is no networking route from the local machine to the scheduler.

**What happened**

With `ECSCluster` for example we see a timeout error when trying to connect to the scheduler.

```python-traceback
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/distributed/comm/core.py", line 286, in connect
    comm = await asyncio.wait_for(
  File "/usr/local/lib/python3.8/asyncio/tasks.py", line 498, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/dask_cloudprovider/aws/ecs.py", line 729, in __init__
    super().__init__(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 281, in __init__
    self.sync(self._start)
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 189, in sync
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 340, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 324, in f
    result[0] = yield future
  File "/usr/local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/usr/local/lib/python3.8/site-packages/dask_cloudprovider/aws/ecs.py", line 926, in _start
    await super()._start()
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 314, in _start
    await super()._start()
  File "/usr/local/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 73, in _start
    comm = await self.scheduler_comm.live_comm()
  File "/usr/local/lib/python3.8/site-packages/distributed/core.py", line 746, in live_comm
    comm = await connect(
  File "/usr/local/lib/python3.8/site-packages/distributed/comm/core.py", line 308, in connect
    raise IOError(
OSError: Timed out trying to connect to tcp://10.53.13.110:8786 after 10 s
```

**What is expected to happen**

We should see an exception, but it shouldn't halt the deletion of the scheduler resource.


Discussion:

Resolving patch:
diff --git a/distributed/core.py b/distributed/core.py
--- a/distributed/core.py
+++ b/distributed/core.py
@@ -43,8 +43,8 @@
 
 class Status(Enum):
     """
-    This Enum contains the various states a worker, scheduler and nanny can be
-    in. Some of the status can only be observed in one of nanny, scheduler or
+    This Enum contains the various states a cluster, worker, scheduler and nanny can be
+    in. Some of the status can only be observed in one of cluster, nanny, scheduler or
     worker but we put them in the same Enum as they are compared with each
     other.
     """
@@ -52,6 +52,7 @@ class Status(Enum):
     closed = "closed"
     closing = "closing"
     closing_gracefully = "closing-gracefully"
+    failed = "failed"
     init = "init"
     created = "created"
     running = "running"
diff --git a/distributed/deploy/spec.py b/distributed/deploy/spec.py
--- a/distributed/deploy/spec.py
+++ b/distributed/deploy/spec.py
@@ -311,7 +311,12 @@ async def _start(self):
             getattr(self.scheduler, "external_address", None) or self.scheduler.address,
             connection_args=self.security.get_connection_args("client"),
         )
-        await super()._start()
+        try:
+            await super()._start()
+        except Exception as e:
+            self.status = Status.failed
+            await self._close()
+            raise RuntimeError(f"Cluster failed to start. {str(e)}") from e
 
     def _correct_state(self):
         if self._correct_state_waiting:
@@ -400,7 +405,7 @@ async def _close(self):
             await asyncio.sleep(0.1)
         if self.status == Status.closed:
             return
-        if self.status == Status.running:
+        if self.status == Status.running or self.status == Status.failed:
             self.status = Status.closing
             self.scale(0)
             await self._correct_state()


Test patch: