Repository: dask/distributed

Issue:
Adaptive can close active workers
I observed in logs some occasions where adaptive scales down the cluster and closes active workers, lengthening the overall computation time.


Looking at the logic for when scaling down can happen: 
* scale down :not all workers are processing, when workers are idle and memory is not an issue

It seems that the semantics of scheduler.idle are not what scheduler.workers_to_close assumes they are:

In the workers_to_close docstring, it says that it will only close workers that are idle (not running anything), yet the code shows that if there is any worker not running anything, it can close any worker marked as idle. 

Workers are marked as idle, however, are not just those workers with no processing: check_idle_saturated marks workers as idle also if they have less than half the average occupancy.

So, in this example:

Three tasks: two with occupancy 4 and one with occupancy 1, running on 4 cores:

* Worker 1 : occupancy 4
* Worker 2 : occupancy 4
* Worker 3 : occupancy 1
* Worker 4 : occupancy 0

In this scenario, both Workers 3 and 4 would be marked as idle and closed.



Discussion:
A change, either in the docstring, or in the operation of `workers_to_close` would be welcome here.
Ah, I see that you're already on this :)

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -1918,7 +1918,7 @@ def workers_to_close(self, memory_ratio=2):
         Find workers that we can close with low cost
 
         This returns a list of workers that are good candidates to retire.
-        These workers are idle (not running anything) and are storing
+        These workers are not running anything and are storing
         relatively little data relative to their peers.  If all workers are
         idle then we still maintain enough workers to have enough RAM to store
         our data, with a comfortable buffer.
@@ -1946,8 +1946,8 @@ def workers_to_close(self, memory_ratio=2):
 
             limit = sum(limit_bytes.values())
             total = sum(worker_bytes.values())
-            idle = sorted(self.idle, key=worker_bytes.get, reverse=True)
-
+            idle = sorted([worker for worker in self.idle if not self.processing[worker]], 
+                          key=worker_bytes.get, reverse=True)
             to_close = []
 
             while idle:


Test patch:
diff --git a/distributed/tests/test_scheduler.py b/distributed/tests/test_scheduler.py
--- a/distributed/tests/test_scheduler.py
+++ b/distributed/tests/test_scheduler.py
@@ -717,6 +717,21 @@ def test_retire_workers(c, s, a, b):
     assert not workers
 
 
+@gen_cluster(client=True, ncores=[('127.0.0.1', 1)] * 4)
+def test_workers_to_close(cl, s, *workers):
+    s.task_duration['a'] = 4
+    s.task_duration['b'] = 4
+    s.task_duration['c'] = 1
+
+    cl.map(slowinc, [1, 1, 1], key=['a-4','b-4','c-1'])
+    while len(s.rprocessing) < 3:
+        yield gen.sleep(0.001)
+
+    wtc = s.workers_to_close()
+    assert all(not s.processing[w] for w in wtc)
+    assert len(wtc) == 1
+
+    
 @gen_cluster(client=True)
 def test_retire_workers_no_suspicious_tasks(c, s, a, b):
     future = c.submit(slowinc, 100, delay=0.5, workers=a.address, allow_other_workers=True)