Repository: dask/distributed

Issue:
Adaptive cannot recover from lost workers
It seems that in the current version of `distributed=2.6.0` (and in older versions also I think), clusters running in adaptive mode cannot (at least, in most cases) recover from workers being unexpectedly lost. This is best exemplified in the case where if all workers are killed at the same time no more workers spin up to replace them, although this issue also persists in more subtle cases. This seems related to both #2990 and #3118.

This can be easily reproduced by the running the following simple example (it currently uses `dask-jobqueue` but I believe this will be true for most `SpecCluster` derived clusters?):

```
import time
from dask_jobqueue import LSFCluster
from dask.distributed import Client


cluster = LSFCluster(cores=1,
                     memory='1gb',
                     walltime='00:05',
                     queue='cpuqueue',
                     env_extra=['conda activate distributed'])

cluster.adapt(minimum_jobs=1, maximum_jobs=2)

client = Client(cluster)
future = client.submit(time.sleep, 60)

client.gather([future])
```

and then killing the one spawned worker before it finishes sleeping. The result is a cluster that remains active, but no new workers are spawned leaving things hanging indefinitely.

Adding some simple log statements (excuse the quick and dirty use of print here):

```
print('Adapt start.', flush=True)
print(f'Target {target}.', flush=True)

print(f'plan - {plan}')
print(f'requested - {requested}')
print(f'observed - {observed}')

print(f'workers - {self.cluster.workers}')
print(f'worker_spec - {self.cluster.worker_spec}')
```

at the top of the  `distributive.deploy.adaptive_core.AdaptiveCore.recommendations` method one can see that the cluster state becomes desynchronised with the scheduler state:

```
Adapt start.
Target 1.
plan - set()
requested - set()
observed - set()
workers - {}
worker_spec - {}
Recommendations {"status": "up", "n": 1}.
Adapt start.
Target 1.
plan - {0}
requested - {0}
observed - set()
workers - {0: <LSFJob: status=running>}
worker_spec - {0: {...}}
Recommendations {"status": "same"}.
.
. <------- The above 8 lines repeat
.
Adapt start.
Target 1.
plan - {0}
requested - {0}
observed - {0}
workers - {0: <LSFJob: status=running>}
worker_spec - {0: {...}}
Recommendations {"status": "same"}.     <---- I kill the worker just after this prints.
Adapt start.
Target 1.
plan - {0}
requested - {0}
observed - set()                        <----- The scheduler no longer sees the worker
workers - {0: <LSFJob: status=running>} <------ The cluster still sees the worker
worker_spec - {0: {...}}
Recommendations {"status": "same"}.
.
. <------- The above 8 lines repeat indefinitely
.
```

namely that the scheduler (which here is the source of `observed` I believe?) correctly detects the worker is no longer alive, but the cluster (which here  is the source of `workers`) does not see this change.

This can seemingly be corrected by setting the `distributed.deploy.lost-worker-timeout` config option to 15s (as per #2990 ):

```
... <----------- Same as previous example
Adapt start.
Target 1.
plan - {0}
requested - {0}
observed - {0}
workers - {0: <LSFJob: status=running>}
worker_spec - {0: {...}}
Recommendations {"status": "same"}.     <---- I kill the worker just after this prints.
Adapt start.
Target 1.
plan - {0}
requested - {0}
observed - set() <------ The scheduler sees the worker has been killed.
workers - {0: <LSFJob: status=running>}
worker_spec - {0: {...}}
Recommendations {"status": "same"}.
.
. <----- The above 8 lines repeat
.
Adapt start.
Target 1.
plan - {0}
requested - set()
observed - set()
workers - {}   <------- The worker is now removed correctly after some time
worker_spec - {0: {...}}
Recommendations {"status": "same"}.
.
. <----- The above 8 lines repeat
.
```

Even though now the cluster `workers` list appears be in sync with the scheduler kept worker list, it seem like this would still not be enough to correct this issue as the 'planned' workers (which are described by the `worker_spec` I think?) match the target number of workers, but the cluster does not request these.

Adding `await self.cluster` to the top of `recommendations` seems to fix this issue whereby the workers are now re-spawned (I think as it leads to `cluster._correct_state` being called?) but I don't know enough about the internals of these classes to know if this is a correct thing to do?

Sorry for the long post!

So three questions -

1) Is it safe to add `await self.cluster` to the top of `recommendations`, or is this a bad solution / just hiding a larger issue?

2) This only works if the `lost-worker-timeout` config is set manually either in code or the config file - should there be a built-in default for this value (apologies if I've missed this!)? 

3) There doesn't seem to be much information about `lost-worker-timeout` - how resilient is it? Is there something missing here which should keep the cluster + scheduler better in sync?


Discussion:

Resolving patch:
diff --git a/distributed/deploy/adaptive.py b/distributed/deploy/adaptive.py
--- a/distributed/deploy/adaptive.py
+++ b/distributed/deploy/adaptive.py
@@ -113,6 +113,14 @@ async def target(self):
             target_duration=self.target_duration
         )
 
+    async def recommendations(self, target: int) -> dict:
+        if len(self.plan) != len(self.requested):
+            # Ensure that the number of planned and requested workers
+            # are in sync before making recommendations.
+            await self.cluster
+
+        return await super(Adaptive, self).recommendations(target)
+
     async def workers_to_close(self, target: int):
         """
         Determine which, if any, workers should potentially be removed from
diff --git a/distributed/deploy/adaptive_core.py b/distributed/deploy/adaptive_core.py
--- a/distributed/deploy/adaptive_core.py
+++ b/distributed/deploy/adaptive_core.py
@@ -178,6 +178,7 @@ async def adapt(self) -> None:
         self._adapting = True
 
         try:
+
             target = await self.safe_target()
             recommendations = await self.recommendations(target)
 


Test patch:
diff --git a/distributed/deploy/tests/test_spec_cluster.py b/distributed/deploy/tests/test_spec_cluster.py
--- a/distributed/deploy/tests/test_spec_cluster.py
+++ b/distributed/deploy/tests/test_spec_cluster.py
@@ -1,5 +1,6 @@
 import asyncio
 import re
+from time import sleep
 
 import dask
 from dask.distributed import SpecCluster, Worker, Client, Scheduler, Nanny
@@ -131,6 +132,35 @@ async def test_scale(cleanup):
         assert len(cluster.workers) == 2
 
 
+@pytest.mark.slow
+@pytest.mark.asyncio
+async def test_adaptive_killed_worker(cleanup):
+    with dask.config.set({"distributed.deploy.lost-worker-timeout": 0.1}):
+
+        async with SpecCluster(
+            asynchronous=True,
+            worker={"cls": Nanny, "options": {"nthreads": 1}},
+            scheduler={"cls": Scheduler, "options": {"port": 0}},
+        ) as cluster:
+
+            async with Client(cluster, asynchronous=True) as client:
+
+                cluster.adapt(minimum=1, maximum=1)
+
+                # Scale up a cluster with 1 worker.
+                while len(cluster.workers) != 1:
+                    await asyncio.sleep(0.01)
+
+                future = client.submit(sleep, 0.1)
+
+                # Kill the only worker.
+                [worker_id] = cluster.workers
+                await cluster.workers[worker_id].kill()
+
+                # Wait for the worker to re-spawn and finish sleeping.
+                await future.result(timeout=5)
+
+
 @pytest.mark.asyncio
 async def test_unexpected_closed_worker(cleanup):
     worker = {"cls": Worker, "options": {"nthreads": 1}}